I am looking to run inference with optimum in a distributed setting with PyTorch (multi-node, multi-CPU/GPU). Is there a recommended approach to do this? My data is coming from a HF Datasets object.
I tried using this solution with the HF Trainer, but it gives me an error when I run it with an optimum model (the optimum model does not have an eval() function).