I am using the summarization example
this is my script for traing using 3 gpus
accelerate launch --config_file='./accelerate.yaml' run_summarization_no_trainer_test.py --seed=42 --preprocessing_num_workers=1 --weight_decay='0.001' --output_dir="samsum_exp/t5/base_3" --per_device_train_batch_size=1 --per_device_eval_batch_size=1 --num_train_epochs=10 --model_name_or_path='t5-base' --num_beams=5 --with_tracking --report_to='wandb' --checkpointing_steps='epoch' --dataset_name="samsum" --source_prefix "summarize: " --gradient_accumulation_steps=2
using 6 gpus
accelerate launch --config_file='./accelerate.yaml' run_summarization_no_trainer_test.py --seed=42 --preprocessing_num_workers=1 --weight_decay='0.001' --output_dir="samsum_exp/t5/base_6" --per_device_train_batch_size=1 --per_device_eval_batch_size=1 --num_train_epochs=10 --model_name_or_path='t5-base' --num_beams=5 --with_tracking --report_to='wandb' --checkpointing_steps='epoch' --dataset_name="samsum" --source_prefix "summarize: " --gradient_accumulation_steps=1
The resuled metrics are as in figure below:
According to this answe It supposed to get identical result but it is not.