Create a simple tokenizer

beannn February 14, 2023, 6:57am 1

I have a preprocessed dataset. The tokens are split by whitespace. So I need a very simple tokenizer to load this. Is there any advice about how to create this?

Topic		Replies	Views
Use a pretrained ByteLevelBPETokenizer on text 🤗Tokenizers	1	3845	July 17, 2020
Save tokenizer with argument 🤗Tokenizers	2	1969	October 26, 2022
Train Retry Tokenizer 🤗Tokenizers	0	224	April 18, 2023
Writing custom tokenizer and wrapping it in tokenizer object 🤗Tokenizers	2	806	June 26, 2023
Can't load tokenizer for 'sshleifer/student_blarge_12_3' 🤗Tokenizers	0	331	May 6, 2021

Create a simple tokenizer

Related topics