[RuntimeError] GPU is required to quantize or run quantize model – Qwen1.5-0.5B-Chat in my Space

[John6666]

It may be possible to use a quantized model in a CPU environment, but it would probably be faster to simply use a non-quantized model in this case.

#MODEL_ID = "Qwen/Qwen1.5-0.5B-Chat-GPTQ-Int4"
MODEL_ID = "Qwen/Qwen1.5-0.5B-Chat"

GPTQ

Loading quantized model on CPU only

Im currently trying to run BloomZ 7b1 on a server with ~31GB available ram. Without quantization loading the model starts filling up swap, which is far from desirable. I tried enabling quantization with load_in_8bit: from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer import torch modelPath = "/mnt/backup1/BLOOM/" device = torch.device("cpu") tokenizer = AutoTokenizer.from_pretrained(modelPath) model = AutoModelForCausalLM.from_pretrained(modelPath, device_map="auto",…