I am trying to use datasets.load_dataset to load multiple big files from disk. I noticed it could merge the content of the files together to be a single dataset. And then I called the datasets.map to tokenize the dataset. My question is: If the total content of the files are much bigger than RAM, …

How to handle big data?

lhoestq May 25, 2023, 3:08pm 7

Topic		Replies	Views
Support of very large dataset? 🤗Datasets	12	10245	August 24, 2022
Working with large datasets 🤗Datasets	5	4080	November 10, 2020
Loading a dataset doesn't actually memory map 🤗Datasets	1	908	September 4, 2023
`load_dataset` results in OOM 🤗Datasets	0	165	June 25, 2024
Trying to figure out when is a dataset stored in memory? 🤗Datasets	4	1133	June 29, 2023