Hi, I’m trying to fine-tune bart-large on 1080Ti (about 11 GB) with deepspeed ZeRO3. But I encountered OOM no matter I used 1 or 4 gpus (with batchsize = 1). I don’t know whether bartlarge is too big for my GPU or I use DeepSpeed incorrectly.
So I tested my code with nothing changed but model from “bart-large” to “bart-base”. I found that the GPU memory usage was the same (about 5000 MiB) no matter I used 1 or 4 gpus. So I guess I use ZeRO3 settings incorrectly. Please help me figure out why. Thank you very much!!!
FYI:
- I use the deep speed configure from transformers/tests/deepspeed/ds_config_zero3.json
- Here is my code
training_args = Seq2SeqTrainingArguments( output_dir='./Model/bartbase', # output directory evaluation_strategy='steps', # Evaluation is done (and logged) every eval_steps. per_device_train_batch_size=train_batch_size, # batch size per device during training per_device_eval_batch_size=2*train_batch_size, # batch size for evaluation gradient_accumulation_steps=update_freq, eval_accumulation_steps=1, max_grad_norm=0.1, # Maximum gradient norm (for gradient clipping). num_train_epochs=20, logging_steps=10, save_steps=save_every, fp16=True, eval_steps=eval_every, disable_tqdm=False, # label_smoothing_factor=label_smoothing, # deepspeed="" ) optimizer = AdamW(model.parameters(), lr=lr, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.01) scheduler = get_polynomial_decay_schedule_with_warmup(optimizer=optimizer, num_warmup_steps=warmup_updates, num_training_steps=total_num_update) trainer = Seq2SeqTrainer( model=model, # the instantiated 🤗 Transformers model to be trained args=training_args, # training arguments, defined above train_dataset=train_dataset, # training dataset eval_dataset=valid_dataset, # evaluation dataset data_collator=collator, tokenizer=tokenizer, optimizers=(optimizer, scheduler) )