Does anyone have working code for training T5-11B on multi-gpu?

🤗Transformers DeepSpeed
1

I have tried so many variations of completely different code and I can’t get it working. I can train any of the smaller t5 models, but once it requires multi-gpu, I can’t get it to work. Tried deepspeed, accelerate, and solutions without using either of those.

So, my question is, does anyone have some code that runs t5-11b for fine tuning? (or know where some is)

2

I’d check these threads:

3

Thanks, I have read through those on previous attempts, but didn’t get them working.

4

Hi, I am also trying to train T5-11b on multi-gpu. Have you succeeded?

5

No, I didn’t get it to work properly.