Im currently trying to run BloomZ 7b1 on a server with ~31GB available ram. Without quantization loading the model starts filling up swap, which is far from desirable. I tried enabling quantization with load_in_8bit: from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer impor…

Loading quantized model on CPU only

vmss2009 January 28, 2024, 2:39pm 4

Same here!!! Can anyone help for the workaround ?

Topic		Replies	Views
How to load quantized LLM to CPU only device Intermediate	0	1953	January 28, 2024
Load quantized model in memory Beginners	1	595	December 8, 2023
SmolVLM 8bit Quantization Problem Models	3	530	November 29, 2024
An error i ve been trying to fix for days now Intermediate	4	462	November 19, 2024
"normal_kernel_cpu" not implemented for 'Char' when trying to import 8-bit model 🤗Transformers	6	1887	February 23, 2025