Character level tokenizer with specific order

I see. It probably works. Unless you want to incorporate it as a new architecture into the code of the Hugging Face library itself, it seems like you can use it from AutoClass by writing tokenizer_config.json and setting trust_remote_code=True.

AutoTokenizer で chiTra トークナイザを読み込む #transformers - Qiita (in Japanese)