GPT-Neo text vs text_target for Seq2Seq Task

Hi there,

I am finetuning a GPT-Neo model on a seq2seq task and am a bit confused about the details on how to fine-tune. Specifially, how to construct and tokenize:

Do I pass input and target to the tokenizer together in tokenizer(text=...) or should I use text for the input and text_target for the target? In [1] it looks like you can just concatenate both and pass to text, but how can the model differentiate between input and target?


[1] gpt-neo-fine-tuning-example/ at main · dredwardhyde/gpt-neo-fine-tuning-example · GitHub