I trained a BPE tokenizer using the wiki-text and now I’m trying to use this tokenizer on a custom dataset from a csv file. What I want to achieve is additional of feature columns in my dataset. But the dataset.map is giving error. [Screenshot 2021-06-23 at 5.47.51 PM]

Custom Dataset with Custom Tokenizer

isarth June 23, 2021, 12:38pm 3

When I train the tokenizer using this Quicktour — tokenizers documentation
The call function is not implemented

Topic		Replies	Views
Tokenizer is not defined 🤗Transformers	5	11247	March 19, 2024
Programmatic way to Tokenization on Custom Text Columns 🤗Tokenizers	0	568	June 27, 2022
How to tokenize using map 🤗Datasets	4	6241	April 14, 2021
Multi class classification Not able to tokenize using map on custom datset Beginners	0	200	August 30, 2021
Save tokenizer with argument 🤗Tokenizers	2	1966	October 26, 2022