Creating dataset slow

[neilhavefun]

I’m trying to create dataset by using .from_generator(). At the beginning the first 1000 items is processed very fast. However, after 1000, the processing speed is extremely slow. I’m curious why this happen? I’m really appreciate if anyone can help me with this problem.

[John6666]

Perhaps the data set has become too large, exceeding the capacity of the RAM, and you are using an SSD or HDD as a substitute for RAM?
You might want to look for some know-how on creating large data sets.

How to create a new large Dataset on disk?

I have my own library which processes corpora of documents much larger than what fits into memory. I would like to create a new HF Dataset by incrementally adding/streaming new examples to it, but because of the number of examples the dataset would not fit into memory. I am not sure if add_item should get used for this as it does not seem to modify the dataset in place but rather returns a new dataset which looks like the dataset would actually be immutable and add_item would create a new data…

Strategy for generating a large dataset

Hi, I have to generate a dataset from 1,000+ large files by: making a random choice with replacement of a file per example (fast, this step takes a total of ~1 min for all examples). We need to keep a list of labels per file that describe some categories the file belongs to. sampling each of the chosen files at a random location (slow, ~a few days) and extracting a numerical vector per example Some constraints: the data is proprietary and the dataset cannot be uploaded online, it has to rem…

[neilhavefun]

Thanks for the reply. Here are some details.
I read data from hdf5 files. And trying to build a hf dataset for training purpose.
The item of my dataset is contained with 3 images and labels.
For instance, pseudo code like
mygen():
files = readfiles(path)
for hdf5_item in files:
with readfile(hdf5) :
yield{{“img1”,img1},{“img2”,img2},{“img3”,img3}}.

The resolutions of imgs are 640X1080X3. It contains larger than 40,000 items.
I would appreciate it if you could provide some concrete examples or ideas for building a dataset like this.

[John6666]

Anyway, I think that using from_generator or from_list will use up too much RAM. If you want to create a very large dataset, you can define a script to load the dataset, but if you can limit the dataset to images, you might be able to use the following method.
In short, you can create a dataset just by uploading it to HF in separate directories.

Create an image dataset

Create a dataset loading script

[neilhavefun]

Thanks for the reply. Is from_generator the most efficient way to create the large-scale dataset? I think the problem is about the RAM. I’m planning to create and save the dataset locally.

[John6666]

I don’t have much experience with datasets libraries, so please consider this as a reference only.
from_ is simple and convenient, but I don’t think it’s suitable for creating large datasets. Unless you add more RAM to your PC…
On the other hand, I think the method of creating on disk and then loading at the end, or writing a script dedicated to loading, is suitable for creating huge datasets.

When reading the dataset, IterableDataset (streaming) should be available.
This saves RAM by reducing the amount of data loaded at once. I don’t know how to use it when creating a dataset.

Stream