How do i batch in streaming of data set

[John6666]

This might be it. The benefit is probably that it will be processed in parallel according to the number of workers?

How do I iterate through <class 'datasets.dataset_dict.IterableDatasetDict'>?

I tried using load_dataset with streaming = Ture for tiiuae/falcon-refinedweb dataset. However, I am unsure how I should iterate through the data. I called the iter() method and next() but I get stopiteration error. TIA!

Streaming batched data

Hello, I’m trying to batch a streaming dataset. This is what I have done so far: coco_train = load_dataset("facebook/pmd", use_auth_token=hf_token, name="coco", cache_dir="/kaggle/working/", streaming=True, split="train") train_ds = coco_train.map(batched=True, batch_size=32) however, doing next(iter(train_ds)) only returns one instance (and it is quite slow). Is there something I’m missing here? How do you get a batch of data?

Now you can iterate on batches using

for batch in train_ds.iter(batch_size=32):
    ...