Any reference on how to train (unsupervised learning) longformer model from scratch (not fine-tuning) using Tensorflow?

Hello folks, i wanna give it a try to use my own dataset to build pre-trained model for longformer, because current pre-trained model only available in English while my task targeting for non-english.

Any reference on how to train longformer model from scratch (not fine-tuning)? Specifically using Tensorflow since i plan to use it for training model for the downstream tasks later.

Thank you in advance!

1 Like