Hi I have a custom dataset of structure: DatasetDict({ train: Dataset({ features: ['sentence', 'label'], num_rows: 12638343 }) test: Dataset({ features: ['sentence', 'label'], num_rows: 1560290 }) validation: Dataset({ features: ['sentence', 'label'], num_rows: 1404261 }) })…

Map method to tokenize raises index error

yaxirhuxxain May 8, 2021, 6:26am 2

I am facing similar issue. Have you found any solution?

2 Likes

Topic		Replies	Views
Map multiprocessing Issue 🤗Datasets	31	17813	July 16, 2024
OverflowError: There was an overflow with type <class 'list'>. Try to reduce writer_batch_size to have batches smaller than 2GB 🤗Datasets	2	2316	May 14, 2022
'i' format requires -2147483648 <= number <= 2147483647 error 🤗Datasets	0	376	January 6, 2023
AttributeError: 'InMemoryTable' object has no attribute '_batches' 🤗Datasets	0	296	February 9, 2024
Dataset map() raises value error when mapping list to dict-like class 🤗Datasets	6	106	August 15, 2024