LM from Scratch for Tensorflow

🤗Transformers
1

Hi,
I was following this tutorial to train a LM from scratch: How to train a new language model from scratch using Transformers and Tokenizers
The result is a pytorch model. Though I need one for Tensorflow. Is there an easy way to convert it?

I tried to modify the training code by using TFTrainer, TFBertForModelLM instead. but TFTrainer is causing trouble with the Data_collator and LineByLineTextDataset objects.
When intitializing the trainer with the data collator i get the error: init() got an unexpected keyword argument ‘data_collator’
When calling trainer.train() (without collator) I receive the error: LineByLineTextDataset object has no attribute ‘_variant_tensor’ .

2

Hi! Here’s a nice example of custom TF MLM learning on XLM-Roberta with Kaggle TPU

https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm

3 Likes
3

Thank you! Will check that link later. Sometimes I wonder why I don’t generally start my research on kaggle :smiley: Has been a helpful source quite a few times.