Currently, there is no GPT2 model that was trained from scratch for Swedish on the hub: Hugging Face – The AI community building the future. . For this project, the goal is to create a strong language generation model for Swedish.
A randomly initialized GPT2 model
A causal language modeling script for Flax is available here. It can be used pretty much without any required code changes.
The desired project output is a GPT2 model that is able to generate Spanish language. A nice generation demo can be created for this.
It might be possible that there is not enough data for the model to perform reasonably well on text generation. In this case, one would have to look at other datasets as well, like
The most important read would be the following colab: