Wav2vec fine-tuning with multiGPU

Models
1

Hi, @patrickvonplaten @valhalla
I’m fine-tuning wav2vec model with Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers at local machine with 4xT4 GPU (16Gb)
I have some problems with training.

  1. Very slowly process
    use_one_gpu
    use_one_gpu1057×316 42.3 KB

use_3_gpu

Why has the learning process slowed down so much?