Why does the falcon QLoRA tutorial code use eos_token as pad_token?

[brando]

@Rocketknight1

However, depending on your fine-tuning task, you may not want the model to learn to predict eos_token at the end of a sequence - if this is the case, simply change the label at that position to the token you do want, or set the label to -100 to mask the label at that position.

Does that answer the questions you had? Feel free to let me know if I missed anything here!

Yes this is what I was going to do because I’m doing fine-tuning for code where syntax matters.

But I need the code. I’ve not had time to write it down. When I do I will share here. To clarify this is what I plan to do:

In the collate function for all seqs in the batch switch the final mask to 1 where the first EOS token is at.