I have tried to register my own tokenization model based on sentencepiece using CustomAITokenizer.register_for_auto_class(āAutoTokenizerā) . But I am falied to do so.
RANITBAG/CustomAItokenizer at main . This the repo link. Can anyone help me in this?