Where to set the Evaluation Batch Size in Trainer

ollibolli · June 17, 2022, 10:04am

Hi, I am training a model on colab with the trainer class. Training is fine. However I am running into the problem that I get a CUDA out of memory error and I am seeing the trainer uses evaluation of batch size = 8 even though I have a batch size of 2.

I did not see a parameter that I can set to change it.
Is my only option to subclass the Trainer class?

Thanks for any tips and help!!

BramVanroy · June 17, 2022, 12:26pm

The default TrainingArguments have the following two options, for training and evaluation respectively. So you’ll need the latter.

github.com

huggingface/transformers/blob/3c7e56fbb11f401de2528c1dcf0e282febc031cd/src/transformers/training_args.py#L141-L144

      
        
            per_device_train_batch_size (`int`, *optional*, defaults to 8):
                The batch size per GPU/TPU core/CPU for training.
            per_device_eval_batch_size (`int`, *optional*, defaults to 8):
                The batch size per GPU/TPU core/CPU for evaluation.

ollibolli · June 17, 2022, 12:56pm

Thank you so much! I was looking through the arguments in the docs but this I have missed! thanks a lot!

Topic		Replies	Views
Trainer will evaluate using my entire validation set(60k), which gives me cuda memory usage issue:. Is there a param that allows evaluating only on some batches in validation set? Beginners	4	936	April 29, 2022
Trainer with adaptive batch size? Beginners	0	508	September 29, 2023
Batch size for trainer.predict() 🤗Transformers	4	5603	November 26, 2022
Eval_batch_size VS per_device_eval_batch_size DeepSpeed	0	567	August 4, 2023
Increasing eval batch size in trainer api causes size mismatch during evaluation 🤗Transformers	0	406	December 24, 2022

Where to set the Evaluation Batch Size in Trainer

Related Topics