I am trying to use datasets.load_dataset to load multiple big files from disk. I noticed it could merge the content of the files together to be a single dataset. And then I called the datasets.map to tokenize the dataset. My question is: If the total content of the files are much bigger than RAM, …

How to handle big data?

lhoestq May 25, 2023, 3:08pm 7

Topic		Replies	Views
Dataset.map hangs on tokenization (relatively small dataset) 🤗Datasets	2	2003	April 22, 2022
Using load_dataset.set_transform() function along with Trainer class 🤗Datasets	4	2627	April 26, 2021
Batching vs. Sharding a Large Dataset 🤗Datasets	4	2248	June 8, 2021
Lazy-Loading binarized shard using Hf-dataset for Hf-Trainer 🤗Datasets	4	2526	June 24, 2021
Support of very large dataset? 🤗Datasets	12	10439	August 24, 2022