Finetuning BioGPT: Encountering Out of Memory error during evaluation

John6666 · April 15, 2025, 6:24am

I found a similar issue, but it seems to be a little different.

Keep getting an OOM when doing a evaluation

opened 10:45PM - 05 Jul 21 UTC

closed 01:51PM - 06 Jul 21 UTC

## Environment info - `transformers` version: 4.8.2 - Platform: Linux-5.…8.0-55-generic-x86_64-with-debian-bullseye-sid - Python version: 3.7.9 - PyTorch version (GPU?): 1.9.0+cu111 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: Yes cuda:0 only one GPU - Using distributed or parallel set-up in script?: No ### Who can help No idea. Models: - albert, bert Library: - trainer: @sgugger ## Information The problem arises when using evaluating a model, I train using a batch size of 64 and 80% of my data but when I do an evaluation (600 000 data points) it ALWAYS give a OOM at 94% even with a batch size of one and the memory allocation IS ALWAYS 3.75 GB no matter the size of the batch. I've deleted the .cache many times and still got the problem. The tasks I am working on is: * [ ] an official GLUE/SQUaD task: (give the name) * [x] my own task or dataset: which is a sequence classification of postal address (max 30 tokens). ## To reproduce My script is base on the [token classification with trainer](https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py).

Topic		Replies	Views
Regarding the eval batch size for large models Models	0	1085	May 9, 2022
Out of memory error when using trainer & output_hidden_states 🤗Transformers	0	695	January 10, 2023
torch.cuda.OutOfMemoryError when evaluate while traning 🤗Transformers	0	502	October 8, 2023
Cuda out of memory error Intermediate	11	40206	January 27, 2025
Trainer will evaluate using my entire validation set(60k), which gives me cuda memory usage issue:. Is there a param that allows evaluating only on some batches in validation set? Beginners	4	1298	April 29, 2022

Finetuning BioGPT: Encountering Out of Memory error during evaluation

Related topics