Big text dataset loading for training

[nitishpandey04]

I want to load a big dataset which is in shards of .txt files. Total dataset size is around 250GB.

If I just load it using load_dataset("text", data_files=[]), then it will load the complete dataset into the memory or it will use Arrow to do memory mapping and reduce the memory footprint of the dataset ?

And If I want to use the dataset for training, is it advisable to directly use Dataset or IterableDataset ?

[John6666]

If you have excess VRAM or RAM, it seems best to use Dataset, but if not, it seems best to use IterableDataset with num_workers= increased.

Best practices for a large dataset

Hi, I have a ~1TB large dataset stored at the HF hub. I can download this on my disk and read it successfully. Nevertheless, it’s large enough that I can’t fit it in my RAM. What is the best practice for training a model on such a dataset? I tried loading the dataset with load_dataset(..., streaming=True) and then having two buffers: one that is being loaded by training process onto GPU and one that, in a separate thread, streams from the dataset and fills up the other buffer. Then, when the i…

How to load VERY LARGE dataset?

[lhoestq]

Hi ! load_dataset() converts the .txt files to arrow and memory maps the arrow file on disk, so yes it doesn’t fill your RAM.

And If I want to use the dataset for training, is it advisable to directly use Dataset or IterableDataset ?

Depends on your dataset size, Dataset is perfectly fine for <1TB. For bigger datasets IterableDataset provides faster initialization (no conversion needed) and saves disk space