Hi folks, I’m trying to play around with the new StarCoder2 model. I got it working with 16bit, but I’m hoping to use this on a machine with only a few GB of memory available, so I’m trying to play with 4bit quantization. I’m just following these instructions . However, I get an import error whenev…

How to load StarCoder2 quantized to 4bits?

CodeHima March 20, 2024, 8:40pm 2

Hello there, I am also experiencing the same issue. Have you come up with a solution?

Topic		Replies	Views
ImportError in bitsandbytes with Accelerate Beginners	3	8340	April 4, 2024
An error i ve been trying to fix for days now Intermediate	4	483	November 19, 2024
AutoModelForCausalLM error with accelerate and bitsandbytes 🤗Accelerate	1	1588	April 15, 2024
Unable to load a pretrained starcoder2 with SFT 🤗Transformers	0	138	April 2, 2024
Loading quantized model on CPU only 🤗Transformers	6	18808	February 3, 2025