TLDR: Hi, I am trying to train a (lora/p-tune) PEFT model on Falcon 40b model using 3 A100s. I am trying to implement model parallelism as bf16/fp16 model wont fit on one GPU. Is there a way to do it? I have implemented a trainer method. According to deepspeed integration documentation , calling th…

Multi GPU training - Model parallelism

stelli February 2, 2024, 12:38am 2

Hi have you figured this out? running into the same error.

Topic		Replies	Views
Model parallel with deepspeed integration Beginners	0	645	September 14, 2021
Model Parallism DeepSpeed	0	186	April 21, 2024
Parallelizing huggingface models DeepSpeed	0	351	July 24, 2023
Manual pipeline parallelization with DeepSpeed DeepSpeed	0	779	January 7, 2023
Model Parallelism and Pipelining for Model Training Beginners	3	3481	April 11, 2024