Accelerator.backward(loss) never done!

🤗Accelerate
1

I am new to accelerate, my code got stuck in accelerator.backward(loss) forever . I used a sample code for finetuning a language model as in the bottom of my question.
Also, the accelerator config is set to have 4 gpus, but it return 1 when I print in the code print(accelerator.num_processes).
Anyone has any idea about my questions?

Thank you all :slight_smile:

from accelerate import Accelerator
import torch
torch.cuda.empty_cache()
from transformers import AutoTokenizer
from datasets import load_dataset

from torch.optim import AdamW
from transformers import get_scheduler
from transformers import AutoModelForSequenceClassification
from torch.utils.data import DataLoader
accelerator = Accelerator()


dataset = load_dataset("yelp_review_full")
dataset["train"][100]
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")


def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)


tokenized_datasets = dataset.map(tokenize_function, batched=True)

tokenized_datasets = tokenized_datasets.remove_columns(["text"])
tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
tokenized_datasets.set_format("torch")
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(100))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(100))
train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=24)
eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)

model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)

optimizer = AdamW(model.parameters(), lr=5e-5)

num_epochs = 1
num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler(
    name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
)

model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(model, optimizer, train_dataloader, eval_dataloader, lr_scheduler)

print(model.device)
from tqdm.auto import tqdm

progress_bar = tqdm(range(num_training_steps))

model.train()
for epoch in range(num_epochs):
    for batch in train_dataloader:

        outputs = model(**batch)
        print(outputs)
        loss = outputs.loss
        print(loss)
        accelerator.backward(loss)
        # loss.backward()

        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        progress_bar.update(1)
2

Ill need the output of accelerate env and the command you’re using to run the code as well, and ideally your version of the library

3

I use accelerate launch sample_bert.py to run and Here is the output of accelerate env:

- `Accelerate` version: 0.16.0
- Platform: Linux-5.15.0-1027-gcp-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Numpy version: 1.21.6
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - dynamo_backend: NO
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 0
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {}
        - fsdp_config: {}
        - megatron_lm_config: {}
        - downcast_bf16: no
4

This needs to be “all” (the default). Youre telling it to just use the first GPU

1 Like