I am trying this tutotial . preprocess function runs fine when I test it with portion of eli5[train]. However it runs I think out of memory when provided whole dataset. Now when I run eli5.map I get name tokenizer is not defined. Could it be memory issue? I modified the preprocess function to return…

Eli5.map :tokenizer not defined

kobosh August 28, 2023, 7:22pm 2

found solution here

Topic		Replies	Views
Tokenizer is not defined 🤗Transformers	5	6334	March 19, 2024
Help defining tokenizer 🤗Tokenizers	0	218	April 28, 2023
Programmatic way to Tokenization on Custom Text Columns 🤗Tokenizers	0	497	June 27, 2022
Create a simple tokenizer 🤗Tokenizers	0	353	February 14, 2023
Tokenizer.train() running out of memory 🤗Tokenizers	0	580	February 9, 2023