How to run 30B meta model on two nodes with accelerate?


I can successfully run the 30B meta model on one node (following load_checkpoint_and_dispatch "Expected all tensors to be on the same device" for > 1 GPU devices 路 Issue #362 路 huggingface/accelerate 路 GitHub). Now I was curious if I can run the same on two nodes to prepare for even larger models. I ran 鈥渁ccelerate config鈥 and 鈥渁ccelerate launch my_script.py鈥 on both nodes, but it seems that the model is just completely loaded on each of the two nodes.

There is nothing to dispatch the model between nodes right now (and it鈥檚 probably too complicated to be added soon).

ok, that鈥檚 what I feared. Thank you very much.