The last time I faced a lot of trouble trying to train a model with Google Colab TPU with PyTorch-XLA. Eventually, I had to give up and force myself to use Tensorflow for their better integration with TPU. I would appreciate it if anyone can share some good resources/tutorials that work to run a Bert Pretraining on PyTorch with Trainer module on Google Colab TPU.
1 Like