Evaluation loss depends on batch size

[ItaiP24]

I train a token classification model on a private dataset.
I noticed that the evaluation loss has different values if I change the value of per_device_eval_batch_size.
Is it a known thing?
The total test set size is not divisible with the batch size (per_device_eval_batch_size*gpu_num).
When I put a batch size that divides the total test set size, I still get a different number from what I get when I calculate the loss directly with torch.nn.CrossEntropyLoss.
Also, accuracy, recall, etc do not change with the batch size.
Thanks for you help!

[John6666]

Apparently a known but neglected issue for several years…

Batch size during training vs batch size during evaluation

I am confused about the difference between batch size during training versus batch size during evaluation. I am trying to measure how batch size influences the inference time(speed of prediction) of different NLP models after they have been trained using the Huggingface Trainer API. The code I used is below def print_summary(result): print(f"Time: {result.metrics['test_runtime']:.2f}") print(f"Samples/second: {result.metrics['test_samples_per_second']:.2f}") print(f"Test Loss: {res…