Create a simple tokenizer

beannn February 14, 2023, 6:57am 1

I have a preprocessed dataset. The tokens are split by whitespace. So I need a very simple tokenizer to load this. Is there any advice about how to create this?

Topic	Replies	Views
Preprocessing data for custom tokenizer 🤗Transformers	209	October 21, 2022
Help defining tokenizer 🤗Tokenizers	215	April 28, 2023
How do you tokenize one long string? Beginners	196	June 24, 2023
Programmatic way to Tokenization on Custom Text Columns 🤗Tokenizers	489	June 27, 2022
How to decode with spaces? 🤗Tokenizers	1320	April 28, 2022

Create a simple tokenizer

Related Topics