I am trying to reduce MT0 to 2-3 languages and afraid of doing it wrong. One problem is i don’t know how tokenizer work with tokenizer.json. There are so many unfamiliar feature like “single” or “pair”. And what is the meaning of number after tokens?
Thank you