Hello I’m going to use the truncation function in my tokenizers.
I know that huggingface cut it to
but I want to cut it not only using
max_length, also using the token I set.
To be, or [DIV] not to be, that is the question
To be, or
not to be, that is the question
Like this! how can I add it?