Hello all
I’m trying to fine-tune GPT2 more or less using the code from that example:
Some things seem slightly outdated and I adapted the code to train with Pytorch-Lightning in a Jupyter notebook. Still im using 99% unchanged code from Github and the same dataset.
Fine-tuning GPT2-medium seems to work. After one epoch the loss is down to roughly 4.
At inference the chatbot only outputs gibberish like for example:
Hello. How are you?
!hey therehow are youwoooowhat are you?wherew where are?do you knowwayokhow are u?tellwhat are uwhatoodoiokwhere dohowi i’mdowhat aredo you?okdo you areyou are ado.you arei doyou arewowi’m so
I don’t understand that.
are there are what?do you?yesdo you?do you?whati amwhat?i.do you have anydodo youokwhatare?yourwhat are what?i see?sohow are youdoisoi’ve anddotoareiidoi’m youidowhat areiok
What do you want to say?
?doidowhatyou are udoi’mdo uaredo uiyou?dodo uiiok,doiokdoi do you aredoare there aredoyouhow arewhat aredodoiwhat uiithat aresodorightwhat?doido u
I tried several settings at inference but it’s mostly similar.
Where do you think it goes wrong? Is the training not working? Over- or underfittig? Or am I making a mistake at inference?
I’m hesitating to post the code yet. Maybe someone of you can already tell if it’s rather about inference or training and I will only post those parts.