How to load this simple audio data set and use dataset.map without memory issues?

lhoestq · December 10, 2024, 1:56pm

we should change from_dict() and make it have it’s own cache directory tbh, people shouldn’t be looking for this unexpected source of OOM. I added a note in the docs at More docs to from_dict to mention that the result lives in RAM by lhoestq · Pull Request #7316 · huggingface/datasets · GitHub

Topic		Replies	Views
Running out of memory during dataset.map() with `AutoFeatureExtractor.from_pretrained("facebook/hubert-large-ls960-ft")` Beginners	3	3621	June 8, 2022
Loading data from Datasets takes too much memory 🤗Datasets	2	567	January 18, 2024
Dataset map during runtime 🤗Datasets	2	1307	September 13, 2023
Misunderstanding around creating audio datasets from Local files 🤗Datasets	12	1771	July 17, 2023
.map - function overloads my Cache Beginners	3	207	August 21, 2023