A streaming dataset's memory footprint continually grows

[John6666]

I think it’s similar to this phenomenon.

Memory leak when streaming

OOM issue with large dataset streaming

For training, we load a multiple datasets (each with 1000+ 1GB+ each arrow files saved using dataset) with load_dataset(…streaming=true) and then interleave them together with interleave_dataset. Even with streaming dataset we are seeing OOM at the interleaving_datasets step. Note no data has been read from datasets yet. Sharding the initial datasets per gpu before interleaving alleviated the memory pressure. Why should there be such memory issues with streaming dataset?