Fine-tuning GPT-J on Articles (Wikipedia)


I鈥檓 using GPT-J (EleutherAI/gpt-j-6B) as a chatbot. To increase the knowledge of the model in specific areas, I would like to fine-tune it on specific data, such as Wikipedia articles (e.g., the Wikipedia page about Marie Curie). For fine-tuning on a TPU, I鈥檓 following the following guide: mesh-transformer-jax/ at master 路 kingoflolz/mesh-transformer-jax 路 GitHub

How can GPT-J be fine-tuned on a Wikipedia article (or a collection of articles)? Can I just provide the plain text file of the Wikipedia article during fine-tuning?

Does anybody have ideas? Any help is highly welcome.