Pretrain GPT-2 from scratch in Mongolian

[bayartsogt]

GPT2 for Mongolian

Goal is to create a strong language generation model for Mongolian
Since initial code and data is pretty much written by @patrickvonplaten and other huggingface members, it should not be so hard to get the first sense.

Model

Randomly initialized GPT2 model

Datasets

We can use OSCAR which is available through datasets

Datasets

A causal language modeling script for Flax is available here. It can be used pretty much without any required code changes.
If there is time left, I’d love to try some private crawling and integrate it datasets.

Expected Outcome

Understandable Mongolian text generation model

Challenges

Lack of data → OSCAR Mongolian is just 2.2G. Maybe we need to research ways to acquire more data with this.

Links

• Colab Notebook to train CLM
• OSCAR
• flax code

[bayartsogt]

I am also so interested in pretraining Mongolian Roberta/DistillBERT on OSCAR and fine-tune it on the translated Mongolian SQuAD2.0.

[bayartsogt]

Great! Please register here.

[patrickvonplaten]

Awesome, officially defining it!