I am looking to run inference with optimum in a distributed setting with PyTorch (multi-node, multi-CPU/GPU). Is there a recommended approach to do this? My data is coming from a HF Datasets object. I tried using this solution with the HF Trainer, but it gives me an error when I run it with an opti…

Recommended Approach for Distributed Inference

philschmid August 1, 2022, 8:34pm 4

Or just python threads. There is no need for torch.distributed. No we don’t have any code example for that.

Topic		Replies	Views
Inference on Multi-GPU/multinode Beginners	4	7582	January 12, 2023
Optimum vs Accelerate 🤗Optimum	5	1181	March 2, 2023
How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? Intermediate	17	18037	September 6, 2023
HF Trainer downstream evaluation on multiple GPUS 🤗Transformers	1	1090	December 21, 2022
Boilerplate for Trainer using torch.distributed Beginners	4	2055	January 11, 2022