Hi, I was wondering if is there a way to download only part of the data of a dataset. In my specific case, I need to download only X samples from oscar English split (X~100K samples). When I try to invoke the dataset builder it asks for >1TB of space so I think it will download the full set of da…

Hi, let me just complete this: split='train[10:20]' returns a slice of the data, but it still downloads everything. If your dataset is too big, please use streaming mode. You can also slice your dataset in streaming mode, see the documentation here: Stream

Download only a subset of a split

🤗Datasets

morenolq April 4, 2022, 4:41pm 3

Thank you, can I also use streaming mode to reach the same?

Download a fraction of data from HuggingFace Datasets

Topic		Replies	Views
Loading just part of dataset 🤗Datasets	4	4133	February 25, 2025
Loading a fraction of data 🤗Datasets	5	4879	May 12, 2023
Downloading a portion of parquet files 🤗Datasets	3	575	May 23, 2024
Streaming in dataset uploads 🤗Datasets	2	28	March 31, 2025
Download only 1 of many parquet file 🤗Datasets	2	165	March 19, 2025

Download only a subset of a split

Related topics