Too many open files on big datasets

[John6666]

I see. Even in that case, there seems to be easier way. Not sure if it would work…

How to load a large hf dataset efficiently?

@lhoestq what do you think about loading this dataset in a lazy way, for example loading it as a batch of 64 samples at a time? Is it more inefficient than loading those shards of data?

Loading methods

ds = load_from_disk('path/to/dataset/directory', keep_in_memory=True) # RAM consumption looks terrible... But won't disk access decrease?