How to concatenate 100s of small datasets into a very large dataset? *Without* loading into memory?

🤗Datasets
1

I have ~200k audio files with associated metadata and features.
These have been stored into ~100 datasets.
Is there a way that I can concatenate these datasets into one without loading them into memory?

2

You can concatenate these datasets without loading them into memory with concatenate_datasets.