I am trying to use datasets.load_dataset to load multiple big files from disk. I noticed it could merge the content of the files together to be a single dataset. And then I called the datasets.map to tokenize the dataset. My question is: If the total content of the files are much bigger than RAM, …

How to handle big data?

lhoestq May 25, 2023, 3:08pm 7

Topic		Replies	Views
Support of very large dataset? 🤗Datasets	12	8600	August 24, 2022
Working with large datasets 🤗Datasets	5	3225	November 10, 2020
Loading a dataset doesn't actually memory map 🤗Datasets	1	456	September 4, 2023
Trying to figure out when is a dataset stored in memory? 🤗Datasets	4	566	June 29, 2023
Caching a dataset with map() when loaded with from_dict() 🤗Datasets	3	2242	March 22, 2023