PreTrain GPT2-Large (and/or GPT2-XL) from scratch in Portuguese

GPT2-Large (and/or GPT2-XL) for Portuguese

Currently, there is no GPT2-Large or GPT2-XL model that was trained from scratch for Portuguese on the hub: Hugging Face โ€“ The AI community building the future. . For this project, the goal is to create a strong language generation model for Portuguese.


A randomly initialized GPT2-Large (and/or GPT2-XL, if we have resources for this :slight_smile:) model


One can make use OSCAR the dataset is also available through the datasets library here: oscar ยท Datasets at Hugging Face.

Available training scripts

A causal language modeling script for Flax is available here. It can be used pretty much without any required code changes.

Desired project outcome

The desired project output is a GPT2 model that is able to generate Portuguese language. A nice generation demo can be created for this.

Links to read upon

The most important read would be the following colab:

1 Like