Finetuning BioGPT: Encountering Out of Memory error during evaluation

John6666 · April 15, 2025, 6:24am

I found a similar issue, but it seems to be a little different.

Keep getting an OOM when doing a evaluation

opened 10:45PM - 05 Jul 21 UTC

closed 01:51PM - 06 Jul 21 UTC

## Environment info - `transformers` version: 4.8.2 - Platform: Linux-5.…8.0-55-generic-x86_64-with-debian-bullseye-sid - Python version: 3.7.9 - PyTorch version (GPU?): 1.9.0+cu111 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: Yes cuda:0 only one GPU - Using distributed or parallel set-up in script?: No ### Who can help No idea. Models: - albert, bert Library: - trainer: @sgugger ## Information The problem arises when using evaluating a model, I train using a batch size of 64 and 80% of my data but when I do an evaluation (600 000 data points) it ALWAYS give a OOM at 94% even with a batch size of one and the memory allocation IS ALWAYS 3.75 GB no matter the size of the batch. I've deleted the .cache many times and still got the problem. The tasks I am working on is: * [ ] an official GLUE/SQUaD task: (give the name) * [x] my own task or dataset: which is a sequence classification of postal address (max 30 tokens). ## To reproduce My script is base on the [token classification with trainer](https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py).

Topic		Replies	Views
Trainer will evaluate using my entire validation set(60k), which gives me cuda memory usage issue:. Is there a param that allows evaluating only on some batches in validation set? Beginners	4	1335	April 29, 2022
Evaluation error: CUDA out of memory 🤗Transformers	0	732	August 22, 2022
Cuda out of memory during evaluation but training is fine 🤗Transformers	12	17430	February 20, 2025
How to process trainer.evaluate in batch mode to deal with Out of Memory error 🤗Datasets	0	333	March 22, 2023
torch.cuda.OutOfMemoryError when evaluate while traning 🤗Transformers	0	515	October 8, 2023

Finetuning BioGPT: Encountering Out of Memory error during evaluation

Related topics