I am using the trainer to train an ASR model, the dataset and the output dimension are huge. This will cause some problems during training. I struggle with it many days, so I post my solution here, hope it can help. compute_metrics out of memory issue during compute_metrics, it will save all the…

Note that for 3, you can have it in your datasets.Dataset computed once and for all with the map method, you just have to store the resuls in a "lengths" column. It will then uses the Dataset features and not try to access every element.

Problems and solution on Trainer

Beginners

voidful December 17, 2021, 6:04pm 4

It should be length instead of lengths, it can be customize using length_column_name now! That’s nice!

Topic		Replies	Views
CUDA out of memory when using Trainer with compute_metrics 🤗Transformers	25	46723	June 25, 2025
WER Metric running out of Memory 🤗Datasets	3	1789	April 30, 2021
How is the eval dataset processed in a trainer? 🤗Transformers	0	515	August 28, 2023
Perhaps your features (`output` in this case) have excessive nesting (inputs type `list` where type `int` is expected) 🤗Transformers	19	693	January 20, 2025
How to process trainer.evaluate in batch mode to deal with Out of Memory error 🤗Datasets	0	332	March 22, 2023

Problems and solution on Trainer

Related topics