I am confused about the difference between batch size during training versus batch size during evaluation. I am trying to measure how batch size influences the inference time(speed of prediction) of different NLP models after they have been trained using the Huggingface Trainer API. The code I used…

Batch size during training vs batch size during evaluation

ndvb August 27, 2023, 12:59pm 2

I also see difference in the evaluation results, when running huggingface trainer, with different evaluation batch size [per_device_eval_batch_size]

1 Like

Topic		Replies	Views
Evaluation loss depends on batch size Beginners	1	164	October 14, 2024
How to correctly measure inference time? Intermediate	0	937	July 25, 2022
Eval_batch_size VS per_device_eval_batch_size DeepSpeed	0	908	August 4, 2023
Batch size in trainer eval loop DeepSpeed	3	4575	April 22, 2022
Evaluation step very slow 🤗Transformers	2	895	July 24, 2025