Does transformers 3.5.1 support auto mixed precision training？

🤗Transformers
1

I’m using a GPTLMHead model in pytorch.
Is it possible , i add autocast() in the forward function in GPTLMHead and change the training process followed the Automatic Mixed Precision — PyTorch Tutorials 1.8.1+cu102 documentation

2

Yes, it’s possible.

3

I train the gpt model using V100 with mixed precision training. But it happens that in the inference time, the generated token is mess with the autocast() open.when i turn off it, everything is ok.