Hello, I am trying to create my custom dataset from musdb18 . It’s a dataset for music source separation. It consists of mp4 files each of them contains 5 waveforms for vocal, bass, drums etc. Here is the script import datasets from pathlib import Path import stempeg import numpy as np _DESCRIPTI…

Memory error while loading custom dataset

sebchw March 3, 2023, 2:06pm 5

Btw. is there a way to set this

datasets.config.DEFAULT_MAX_BATCH_SIZE = 10

somewhere in the load_dataset function? I can’t see it.

1 Like

Topic		Replies	Views
Proprietary database load error: TypeError: Argument 'storage' has incorrect type (expected pyarrow.lib.Array, got pyarrow.lib.ChunkedArray) 🤗Datasets	2	1137	January 25, 2022
How to save/use only the first 20k samples of a dataset 🤗Datasets	1	64	December 23, 2024
Common Voice 8.0.0 en using all available RAM 🤗Datasets	7	907	August 5, 2022
GeneratorBasedBuilder gets stuck & consumes all RAM 🤗Datasets	2	787	February 8, 2022
Arrowmemoryerror: realloc of size 32 GB failed 🤗Datasets	2	3255	January 6, 2023