Hi im currently trying to map about 5000 audio files into my dataset in aws Sagemaker. When i try to do that, my Kernel crashes after a few minutes, i assume that the its overloading my chache memory… When i use map with only 1000 files it works fine! Does anybody know what i can do about that? That…

.map - function overloads my Cache

lhoestq August 21, 2023, 1:50pm 4

Yup the data collator can take a batch as input and apply the padding

I don’t have a code example but you can check the DataCollatorWithPadding that does tokenization + padding for example: Data Collator

1 Like

Topic		Replies	Views
Running out of memory during dataset.map() with `AutoFeatureExtractor.from_pretrained("facebook/hubert-large-ls960-ft")` Beginners	3	3623	June 8, 2022
How to load this simple audio data set and use dataset.map without memory issues? 🤗Datasets	12	4322	December 10, 2024
Mapping large datasets 🤗Datasets	4	529	February 15, 2022
Multiprocessing map taking too much memory footprint 🤗Datasets	17	5902	April 5, 2024
Padding in datasets 🤗Datasets	6	5080	October 21, 2021