I am trying this tutotial . preprocess function runs fine when I test it with portion of eli5[train]. However it runs I think out of memory when provided whole dataset. Now when I run eli5.map I get name tokenizer is not defined. Could it be memory issue? I modified the preprocess function to return…

Eli5.map :tokenizer not defined

kobosh August 28, 2023, 7:22pm 2

found solution here

Topic		Replies	Views
Tokenizer.train() running out of memory 🤗Tokenizers	0	751	February 9, 2023
Map multiprocessing Issue 🤗Datasets	31	17662	July 16, 2024
Tokenizer is not defined 🤗Transformers	5	11191	March 19, 2024
[SOLVED] Dataset.map() is frozen on ELI5 🤗Datasets	2	2261	August 24, 2020
Why do I get this error running tokenizer? Beginners	6	17692	August 20, 2020