Hey,
I am wondering if simply adding a list to the special tokens definition will allow the model to ignore certain tokens. In our dataset we have certain tokens that are repetitive and have no meaning, so I would like the model to simply ignore these. If i added this to the special tokens list when making a tokenizer, would this work as I intend it to?