Deepspeed inference and infinity offload with bitsandbytes 4bit loaded models

[AvivB]

I am getting this error too.

Edit: There appears to be support now:

DeepSpeed Mixture-of-Quantization (MoQ)