Hugging Face Forums
Creating tokenizer from counts file?
🤗Tokenizers
sabharadwaj
February 9, 2023, 7:37pm
1
I would like to train a wordpiece tokenizer from scratch from a counts file with tokens and counts.