Hi, I’m new to HF dataset and I tried to create datasets based on data versioned in lakeFS (MinIO S3 bucket as storage backend) Here I’m using ±30000 PIL image from MNIST data however it is taking around 12min to execute, which is a lot! From what I understand, it is loading the images into cache…

> Update The bottleneck, from what I understand, was making one network request per file For 30k images, this meant 30k separate GET requests to the MinIO server through the S3 API, and that was killing the performance Using webDataset to transform the large number of files to few .tar files and p…

Creating a HF Dataset from lakeFS with S3 storage takes too much time!

🤗Datasets

system Closed June 24, 2025, 12:37am 9

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Load_datasets is extremely slow in loading HF datasets Beginners	1	2415	December 15, 2023
Extremely slow data loading of imagefolder 🤗Datasets	9	2404	January 4, 2024
Creating dataset slow 🤗Datasets	5	119	December 18, 2024
Loading Huge Image Dataset seems to take a lot of time 🤗Datasets	7	3729	May 16, 2022
Accessing dataset is very slow compared to torchvision 🤗Datasets	2	1310	May 24, 2022

Creating a HF Dataset from lakeFS with S3 storage takes too much time!

Related topics