Where to set the Evaluation Batch Size in Trainer

ollibolli · June 17, 2022, 10:04am

Hi, I am training a model on colab with the trainer class. Training is fine. However I am running into the problem that I get a CUDA out of memory error and I am seeing the trainer uses evaluation of batch size = 8 even though I have a batch size of 2.

I did not see a parameter that I can set to change it.
Is my only option to subclass the Trainer class?

Thanks for any tips and help!!

BramVanroy · June 17, 2022, 12:26pm

The default TrainingArguments have the following two options, for training and evaluation respectively. So you’ll need the latter.

github.com

huggingface/transformers/blob/3c7e56fbb11f401de2528c1dcf0e282febc031cd/src/transformers/training_args.py#L141-L144

      
        
            per_device_train_batch_size (`int`, *optional*, defaults to 8):
                The batch size per GPU/TPU core/CPU for training.
            per_device_eval_batch_size (`int`, *optional*, defaults to 8):
                The batch size per GPU/TPU core/CPU for evaluation.

ollibolli · June 17, 2022, 12:56pm

Thank you so much! I was looking through the arguments in the docs but this I have missed! thanks a lot!

Topic		Replies	Views
Trainer will evaluate using my entire validation set(60k), which gives me cuda memory usage issue:. Is there a param that allows evaluating only on some batches in validation set? Beginners	4	1205	April 29, 2022
Colab error (memory crashes) Beginners	3	3002	April 22, 2021
Trainer with TPUs Beginners	3	2547	April 13, 2022
Batch size in trainer eval loop DeepSpeed	3	4252	April 22, 2022
Trainer with adaptive batch size? Beginners	0	879	September 29, 2023

Where to set the Evaluation Batch Size in Trainer

Related Topics