@mrm8488, this may not be exactly what you’re looking for but I have been finetuning GPT2 with a language model head using my own loss function. I tried to detail in this post as much of the implementation as possible. Maybe parts of it will be useful for you as well since I am also using Trainer.
1 Like