I have the following simple code copied from Huggingface examples: model_checkpoint = "distilgpt2" from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True) def tokenize_function(examples): return tokenizer(examples["text"]) from datase…

Huggingface Dataset.map shows red progress bar when batched=True

mariosasko October 24, 2022, 3:51pm 2

Hi! You can find the explanation and solution to this issue in this thread: While tokenizing the dataset I'm getting this type of error

Topic		Replies	Views
HuggingFace dataset: each element in list of batch should be of equal size 🤗Datasets	3	8419	August 10, 2023
Loading a dataset doesn't actually memory map 🤗Datasets	1	473	September 4, 2023
Trouble batch mapping dataset to tokenizer 🤗Datasets	1	376	June 12, 2023
The datasets.map() method doesn't keep tensor format from `tokenizer` 🤗Datasets	1	1298	November 4, 2022
Why use batched=True in map function? 🤗Datasets	2	3895	May 17, 2022