OpenAI: how to spread multiple environments on multiple GPUs using Huggingface?

Dear all,

I would like to implement some reinforcement learning algorithms and for that I’m using Open AI Gymnasium.
In some algorithms I need to generate multiple parallel models using PyTorch: one model runs on a single GPU and the network weights are updated asynchrounosly.

Since I could use Huggingface only for supervised task (like image classification), do you have any hint, on how to use Huggingface for RL environments?