Hi,
I was wondering if is there a way to download only part of the data of a dataset.
In my specific case, I need to download only X samples from oscar English split (X~100K samples).
When I try to invoke the dataset builder it asks for >1TB of space so I think it will download the full set of data at the beginning.
Hello 
You can load a part of split by slicing:
train_10_20_ds = datasets.load_dataset('bookcorpus', split='train[10:20]')
You can refer to more ways of slicing and loading here.
Thank you, can I also use streaming mode to reach the same?
Hi, let me just complete this: split='train[10:20]' returns a slice of the data, but it still downloads everything.
If your dataset is too big, please use streaming mode. You can also slice your dataset in streaming mode, see the documentation here: Stream
Thank you, that’s the case indeed.
When I stream a dataset that’s too big, it always seems to get stuck after certain point (program hanging and not making progress), like after 10000 samples etc. After a long time it says something like reconnect to data host. Any reasons for that? Thanks!
It depends on the host. Some datasets are hosted on HF, but some others have their data files hosted on the original dataset author/platform. You can check how the dataset is loaded by checking its repository on HF. Which dataset did you try to load ?