Im currently trying to run BloomZ 7b1 on a server with ~31GB available ram. Without quantization loading the model starts filling up swap, which is far from desirable. I tried enabling quantization with load_in_8bit: from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer impor…

Loading quantized model on CPU only

gouse-73 December 12, 2023, 9:08am 3

Did you guys got any solution for loading model only on CPU with quantization?

1 Like