How do I release memory after using AutoModel.from_pretrained() to load model

[Arcolaus]

after execute following code on apple’s silicon chip , even though the program ends memory is still occupied too much.
sorry i’m a beginner of both English and Deep learning. Is it a bug or just I used a wrong way?

[John6666]

I generally do it like this, but it’s probably not right and there are smarter ways to do it. I’m interested too.

import torch
import gc
del llm_model
torch.cuda.empty_cache()
gc.collect()

[Arcolaus]

thank you, actully del llm_model works, the problem is the memory monitor didn’t refresh

[John6666]

Hmmm, like this?
In your case anyway, it would be better to remove it from VRAM first and then delete it.

But well, the torch tensor used in the model’s content is quite stubborn…
If you don’t delete the tensors in the model, not the pipes or the model, one by one, they might be referenced from somewhere and won’t disappear.

import torch
import gc
llm_model.to("cpu")
del llm_model
torch.cuda.empty_cache()
gc.collect()

It is more reliable to del together any relevant models and pipes.

[Arcolaus]

thank you! it’s helpful