I found 🤗 Transformers Notebooks — transformers 4.12.0.dev0 documentation and then Google Colab .
The notebook will create examples which have the same text in the input and the labels. What is the purpose of such a model? Is it training some autoencoder task? I would think a more interesting challenge would be: Given input sample of text, have the label be the continuation of the sample of text.