Hello
I鈥檓 using GPT-J (EleutherAI/gpt-j-6B) as a chatbot. To increase the knowledge of the model in specific areas, I would like to fine-tune it on specific data, such as Wikipedia articles (e.g., the Wikipedia page about Marie Curie). For fine-tuning on a TPU, I鈥檓 following the following guide: mesh-transformer-jax/howto_finetune.md at master 路 kingoflolz/mesh-transformer-jax 路 GitHub
How can GPT-J be fine-tuned on a Wikipedia article (or a collection of articles)? Can I just provide the plain text file of the Wikipedia article during fine-tuning?