LoRa fine tuning a chatbot on 6GB VRAM GPU

Hello, I am doing a chatbot working with data from IMDB.

I was thinking of using meta-llama/Llama-3.2-1B-Instruct and LoRa fine tuning it on BrightData/IMDb-Media Dataset. I have a 6GB VRAM GPU and not much time.

My questions are: Is this the right model to use for the task, or should I use a smaller one? Is it good paired with the dataset? Will 6GB VRAM be enough for LoRa fine tuning it? Also, should I split the dataset and LoRa tune it on a smaller scale? The dataset has 250K rows.

1 Like

I’m not very familiar with it, but training requires a lot of VRAM, so even if you use LoRA with fp16 precision at 1B, it might be just about possible with 6GB of VRAM.
Well, I think it won’t crash if you have enough RAM…
If you can use QLoRA, you can save a lot of VRAM, but I’ve never used it.:sweat_smile: