For training, we load a multiple datasets (each with 1000+ 1GB+ each arrow files saved using dataset) with load_dataset(…streaming=true) and then interleave them together with interleave_dataset. Even with streaming dataset we are seeing OOM at the interleaving_datasets step. Note no data has been …

OOM issue with large dataset streaming

John6666 March 6, 2025, 4:06pm 2

Is the data set file itself too large…?
If it seems like a bug, let’s call lhonestq.

Topic		Replies	Views
How to load a large hf dataset efficiently? 🤗Datasets	5	2197	January 22, 2024
Streaming in dataset uploads 🤗Datasets	2	40	March 31, 2025
"too many open files" despite streaming with IterableDataset 🤗Datasets	2	25	January 27, 2025
Using large dataset with accelerate 🤗Accelerate	0	32	March 6, 2025
Recommended max size of dataset? 🤗Datasets	5	66	March 11, 2025