Hello, I am trying to create my custom dataset from musdb18 . It’s a dataset for music source separation. It consists of mp4 files each of them contains 5 waveforms for vocal, bass, drums etc. Here is the script import datasets from pathlib import Path import stempeg import numpy as np _DESCRIPTI…

Memory error while loading custom dataset

sebchw March 3, 2023, 1:58pm 4

@polinaeterna
However, solution you provided looks very promising. If I do this that way will it write stem files as bytes to an arrow file or these will be searched on disk while decoding?

Topic		Replies	Views
Proprietary database load error: TypeError: Argument 'storage' has incorrect type (expected pyarrow.lib.Array, got pyarrow.lib.ChunkedArray) 🤗Datasets	2	1137	January 25, 2022
How to save/use only the first 20k samples of a dataset 🤗Datasets	1	64	December 23, 2024
Common Voice 8.0.0 en using all available RAM 🤗Datasets	7	907	August 5, 2022
GeneratorBasedBuilder gets stuck & consumes all RAM 🤗Datasets	2	787	February 8, 2022
Arrowmemoryerror: realloc of size 32 GB failed 🤗Datasets	2	3255	January 6, 2023

Memory error while loading custom dataset

Related topics