Why transformers doesn't use Multiple GPUs (to increase tokens per second)?

[John6666]

Possibly:

CUDA Memory issue for model.generate() in AutoModelForCausalLM

Hello everyone, I have 4 A100 GPUs and I’m utilizing Mixtral with dtype set as bfloat16 for a text generation task on these GPUs. I’m aware that by using device_map="balanced_low_0", I can distribute the model across GPUs 1, 2, and 3, while leaving GPU 0 available for the model.generate() function, as detailed in the documentation here: [Handling big models for inference]. The Mixtral model with bfloat16 type occupies approximately 90 GB, which is loaded onto GPUs 1, 2, and partially on GPU 3.…

@victor on Hugging Face: "🙋 Calling all Hugging Face users! We want to hear...