Hi there,
I am finetuning a GPT-Neo model on a seq2seq task and am a bit confused about the details on how to fine-tune. Specifially, how to construct and tokenize:
Do I pass input and target to the tokenizer together in tokenizer(text=...) or should I use text for the input and text_target for the target? In [1] it looks like you can just concatenate both and pass to text, but how can the model differentiate between input and target?
Thanks
[1] gpt-neo-fine-tuning-example/gpt_neo.py at main · dredwardhyde/gpt-neo-fine-tuning-example · GitHub