[RuntimeError] GPU is required to quantize or run quantize model – Qwen1.5-0.5B-Chat in my Space

Beginners
1

Hello everyone😊,
I’d like to test the model on the free CPU environment—do you have any suggestions?

I’m encountering an error when trying to deploy the Qwen1.5-0.5B-Chat model in my Hugging Face Space running on CPU-only (free) .

MyQwen1.5 0.5B Chat - a Hugging Face Space by funme

Thank you :grinning_face:
Here the full log: tokenizer_config.json: 0%| | 0.00/1.29k [00:00<?, ?B/s]
tokenizer_config.json: 100%|██████████| 1.29k/1.29k [00:00<00:00, 7.24MB/s]
vocab.json: 0%| | 0.00/2.78M [00:00<?, ?B/s]
vocab.json: 100%|██████████| 2.78M/2.78M [00:00<00:00, 27.1MB/s]
merges.txt: 0%| | 0.00/1.67M [00:00<?, ?B/s]
merges.txt: 100%|██████████| 1.67M/1.67M [00:00<00:00, 31.1MB/s]
tokenizer.json: 0%| | 0.00/7.03M [00:00<?, ?B/s]
tokenizer.json: 100%|██████████| 7.03M/7.03M [00:00<00:00, 58.3MB/s]
config.json: 0%| | 0.00/1.26k [00:00<?, ?B/s]
config.json: 100%|██████████| 1.26k/1.26k [00:00<00:00, 7.28MB/s]
Traceback (most recent call last):
File “/home/user/app/app.py”, line 9, in
model = AutoModelForCausalLM.from_pretrained(
File “/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py”, line 571, in from_pretrained
return model_class.from_pretrained(
File “/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py”, line 309, in _wrapper
return func(*args, **kwargs)
File “/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py”, line 4389, in from_pretrained
hf_quantizer.validate_environment(
File “/usr/local/lib/python3.10/site-packages/transformers/quantizers/quantizer_gptq.py”, line 65, in validate_environment
raise RuntimeError(“GPU is required to quantize or run quantize model.”)
RuntimeError: GPU is required to quantize or run quantize model.

1 Like
2

It may be possible to use a quantized model in a CPU environment, but it would probably be faster to simply use a non-quantized model in this case.

#MODEL_ID = "Qwen/Qwen1.5-0.5B-Chat-GPTQ-Int4"
MODEL_ID = "Qwen/Qwen1.5-0.5B-Chat"
3

Thank you😊 , I need a model size smaller than 700 MB , I’m going to change model, if I can’t use this model

1 Like
4

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.