Hello, I am a beginner to llm and I was trying to use Qlora to fine tune the llama3-7B model. I was dealing with a text classification problem, so I used AutoModelForSequenceClassification(model_name, quantization_config=bnb_config, num_labels=100), and I use deepspeed zero2 to reduce the memory used in each gpu. When I removed the quantization_config in AutoModelForSequenceClassification, the code runs perfectly, but when I add this parameter, the error goes like: Expected all tensors on the same device. Does anyone know what is going on there? Thanks for your time!
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! | 1 | 861 | September 20, 2023 | |
| RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select) | 5 | 3257 | August 26, 2024 | |
| RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! I am on a single T4 GPU | 6 | 909 | June 10, 2024 | |
| Bitsandbytes quantization and QLORA fine-tuning | 1 | 118 | November 5, 2024 | |
| BitsAndBytes With DDP | 3 | 36 | October 7, 2024 |