In the documentation for T5 eos_token, it is pointed out that:
-
eos_token (
str, optional, defaults to"</s>") — The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the sep_token.
But there doesn’t seem to be anywhere I can set this sep_token