How do i load part of the data set

[sKlklkjjkj]

I have big dataset un_pc

I want to load part like 200 k rows for start how do i do this

[John6666]

I wonder if streaming would be a good option…

Stream

Best practices for a large dataset

Hi, I have a ~1TB large dataset stored at the HF hub. I can download this on my disk and read it successfully. Nevertheless, it’s large enough that I can’t fit it in my RAM. What is the best practice for training a model on such a dataset? I tried loading the dataset with load_dataset(..., streaming=True) and then having two buffers: one that is being loaded by training process onto GPU and one that, in a separate thread, streams from the dataset and fills up the other buffer. Then, when the i…

How to load a large hf dataset efficiently?

It’s generally a good idea indeed if you want to save disk space or to not have to wait to download the full dataset. For example you can stream the dataset using the datasets library, by passing streaming=True to load_dataset(). However even in streaming mode, you better have multiple shards in order to do parallel streaming. And ideally use file formats that work well with streaming like WebDataset or Parquet. @aaditya did try streaming mode on this dataset but the custom loading script of …

[Victorano]

First, load the dataset from hf, then you can select how much rows you need.
For example:

from datasets import load_dataset
dataset_name = "Helsinki-NLP/un_pc"
dataset = load_dataset(dataset_name, split="train")
train_dataset = dataset.select(range(200000))

Note that the full dataset will be downloaded on your computer but only the selected 200k rows will be the train_dataset value

[Zelgodiz]

You can load a subset of your dataset in Hugging Face using the load_dataset() function with filtering options. Here are a few ways to do it:

1. Using .select() to Load a Specific Number of Rows

If your dataset is already loaded, you can select 200,000 rows like this:

from datasets import load_dataset

dataset = load_dataset("your_dataset_name")
subset = dataset["train"].select(range(200000))  # Select first 200k rows

2. Using Streaming to Avoid Full Download

If your dataset is too large to fit in memory, use streaming mode:

dataset = load_dataset("your_dataset_name", split="train", streaming=True)
subset = dataset.take(200000)  # Load only 200k rows

This prevents downloading the entire dataset at once.

3. Using data_files to Load a Specific File

If your dataset consists of multiple files, you can load only a specific portion:

dataset = load_dataset("your_dataset_name", data_files={"train": "train_part1.csv"})

For more details, check out the Hugging Face documentation or community discussions. Let me know if you need help with a specific dataset!