I am trying to use datasets.load_dataset to load multiple big files from disk. I noticed it could merge the content of the files together to be a single dataset. And then I called the datasets.map to tokenize the dataset. My question is: If the total content of the files are much bigger than RAM, …

How to handle big data?

lhoestq May 25, 2023, 3:08pm 7

Topic		Replies	Views
Support of very large dataset? 🤗Datasets	12	10290	August 24, 2022
Big text dataset loading for training 🤗Datasets	2	42	May 7, 2025
Working with large datasets 🤗Datasets	5	4101	November 10, 2020
Loading a dataset doesn't actually memory map 🤗Datasets	1	921	September 4, 2023
`load_dataset` results in OOM 🤗Datasets	0	169	June 25, 2024