How calculate loss.backward() in MLM by PyTorch

Models
1

I am going to train an MLM model by pytorch, but in the training part, I do not know how to calculate the
loss.backward.

model_name = "distilbert-base-uncased"

model = AutoModelForMaskedLM.from_pretrained(model_name, max_length=256,  return_dict=True)

model.train()
device = torch.device("cpu")

for epoch in range(epochs):
    loop = tqdm(dataloader)
    for batch in loop:
        optimizer.zero_grad()
        
        input_ids = batch['input_ids'].to(device)
        labels = batch['labels'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        
        loss = outputs.loss
        loss.backwards()
        optimizer.step()

@lewtun

2

Hey @MahdiA what you have looks pretty close - I think you just need loss.backward() instead of loss.backwards()

Does that solve your issue?

1 Like