Hello I’m going to use the truncation function in my tokenizers.
I know that huggingface cut it to max_length option,
but I want to cut it not only using max_length, also using the token I set.
ex)
To be, or [DIV] not to be, that is the question
→
To be, or
not to be, that is the question
Like this! how can I add it?