Recently, I want to fine-tuning Bart-base with Transformers (version 4.1.1). The fine-tuning process is very smooth with compute_metrics=None in Trainer. However, when I implement a function of computing metrics and offer this function to Trainer, I received the CUDA out of memory error during the evaluation stage.
I want to try this feature, so my implementation is straightforward.
Because when compute_metrics=None, the training process is normal, so I think it can’t be the problem of batch size. But I still tried a smaller batch size, but even if I set the batch size to 1, the situation is the same. I even try tinier-bart, but I received the same error at last.
One thing caught my attention . When I set a tiny batch size, the memory will not fill up at once, but the occupancy rate has increased until the memory can’t hold more data. That is, the processed data is not released in time. Is there any magic operation to solve this problem?
When computing metrics inside the Trainer, your predictions are all gathered together on the device (GPU/TPU) and only passed back to the CPU at the end (because that operation can be slow). If your dataset is large (or your model outputs large predictions) you can use eval_accumulation_steps to set a number of steps after which your predictions are sent back to the CPU (slower but uses less device memory). This should avoid your OOM.
This error means you are trying to get predictions that just don’t fit in RAM, so there is nothing Trainer can do to help. I don’t know which bart models you’re using, but it looks like you have huge logits so you should split your evaluation dataset in small parts or use a custom evaluation loop.
Hi, I’m still struggling with this issue. I’m trying to finetune a Bart model and while I can get it to train, I always run out of memory during the evaluation phase. This does not happen when I don’t use compute_metrics, so I think there’s an issue there - when I don’t use compute_metrics I can run batch sizes of up to 16, however on using compute metrics, I can’t even use a batch size of 1 even with eval accumulation.
Could you please explain why compute metrics is so much heavier when I can run training and evaluation without issues otherwise? In your answer above you mentioned that the trainer holds all predictions on the GPU but why is this being done for metrics calculation?
I have used Fairseq for seq2seq tasks with similarly sized models before this, and have never run into this issue before, so I was also wondering if they do metrics computation differently.
I also experience this when including my own compute_metrics implementation, and it gradually increases GPU memory occupation over time.
Could it be that data structures (tensors I assume) used in our own implementation with each estimation are filling up GPU space and this is overloading our GPU device, and somehow default implementation is using memory garbage collector better? Should we somehow dump memory from variables not being used anymore over time? It seems variables are not being dumped after the compute_metrics() function is done? @sgugger
i was getting this same error so now i am doing this eval_accumulation_steps=16 with per_device_eval_batch_size=1, but i am now getting error “your ram collapsed bcoz you have used up the available ram” in google colab, any more help will be appreciated, i am using colab pro with 15 GB available GPU and model size 2.12 GB Pegasus,dataset Dialogsum, my train batch size is also 2
I figured it out, maybe. Whatever is returned in preprocess_logits_for_metrics(logits, labels) will be available in pred.predictions inside compute_metrics(pred). By returning (pred_ids, labels), pred.predictions becomes a tuple, where the second element is the labels. But the labels will still be available in pred.label_ids no matter what. So only the predictions need to be returned by preprocess_logits_for_metrics, and in that case, we don’t need to access the predicitons at index 0. As for only performing argmax on logits, I still don’t understand. Would that not just take the first element in the batch, and ignore the rest?
I think your understanding is correct on both. logits would only take the first sequence in the batch assuming all other training args are left to their defaults – I modified above snippet to remove the subscript.
Sorry for the late reply, Actually I used logits because the version I was using was passing both logits and labels in logits (maybe I was also misinterpreting them). I agree with you, the correct snippet should be with logits instead of logits in the argmax.