Does transformers 3.5.1 support auto mixed precision trainingļ¼Ÿ

šŸ¤—Transformers
1

Iā€™m using a GPTLMHead model in pytorch.
Is it possible , i add autocast() in the forward function in GPTLMHead and change the training process followed the Automatic Mixed Precision ā€” PyTorch Tutorials 1.8.1+cu102 documentation

2

Yes, itā€™s possible.

3

I train the gpt model using V100 with mixed precision training. But it happens that in the inference time, the generated token is mess with the autocast() open.when i turn off it, everything is ok.