I am trying this tutotial . preprocess function runs fine when I test it with portion of eli5[train]. However it runs I think out of memory when provided whole dataset. Now when I run eli5.map I get name tokenizer is not defined. Could it be memory issue? I modified the preprocess function to return…

Eli5.map :tokenizer not defined

kobosh August 28, 2023, 7:22pm 2

found solution here

Topic		Replies	Views
[SOLVED] Dataset.map() is frozen on ELI5 🤗Datasets	2	2286	August 24, 2020
Why do I get this error running tokenizer? Beginners	6	18003	August 20, 2020
Can't load tokenizer for 'sshleifer/student_blarge_12_3' 🤗Tokenizers	0	337	May 6, 2021
How to tokenize using map 🤗Datasets	4	6351	April 14, 2021
Map method to tokenize raises index error 🤗Datasets	9	4299	June 9, 2021