How to add additional custom pre-tokenization processing?

[shtoshni]

Hi,

I was able to create a Custom Pretokenizer based on the example linked above. But I’m not able to save the tokenizer due to the exception “Custom PreTokenizer cannot be serialized”. I’m wondering how to bypass this.