I’m part of the BirdSet team, and we’ve identified an issue with our current Builder script . Some of the audio datasets we work with are quite large, and we aim to provide access to individual audio files. To achieve this, we first download the archive file, extract its contents, and then generate …

Optimizing Disk Usage for Large (Audio) Datasets

lhoestq November 30, 2024, 3:54pm 5

Hi ! if you load_dataset() a AudioFolder formatted dataset it won’t double the storage (the arrow table of the dataset will just contain links to the audio files on your disk)

1 Like

Topic		Replies	Views
Understanding the `Datasets` cache system 🤗Datasets	2	3304	May 19, 2023
BuilderScript cleanup during extract of archives 🤗Datasets	0	66	November 14, 2024
Not able to use Custom Speech Data for training ASR 🤗Datasets	2	320	September 20, 2023
Can Data Files be generated upon dataset load? Beginners	3	454	March 4, 2022
Dataset loading script for an audio dataset 🤗Datasets	5	673	September 2, 2022

Optimizing Disk Usage for Large (Audio) Datasets

Related topics