I am trying this tutotial . preprocess function runs fine when I test it with portion of eli5[train]. However it runs I think out of memory when provided whole dataset. Now when I run eli5.map I get name tokenizer is not defined. Could it be memory issue? I modified the preprocess function to return…

Eli5.map :tokenizer not defined

kobosh August 28, 2023, 7:22pm 2

found solution here

Topic		Replies	Views
Tokenizer is not defined 🤗Transformers	5	10555	March 19, 2024
[SOLVED] Dataset.map() is frozen on ELI5 🤗Datasets	2	2226	August 24, 2020
Help defining tokenizer 🤗Tokenizers	0	280	April 28, 2023
Error in Dataset Map Function Beginners	3	2174	March 22, 2023
Map with tokenize function stuck in the beginning 🤗Datasets	4	47	December 27, 2024