I’ve tried a few different approaches to iterate on 1M records with datasets but all seem quite slow . So this time I’m trying the recommended streaming method from the recent post but still it’s extremely slow. It seems for dataset there will always be multiple ways to iterate but most (or all) of t…

Iterating on dataset extremely slow

Manu101 September 27, 2024, 4:06pm 7

Hi @zhh210 , can you please provide the exact code you substituted for Ray ? And how much the difference is between the two approaches?

Thanks

Topic		Replies	Views
Roadmap/timeline for dataset streaming 🤗Datasets	9	2281	July 5, 2021
Parsing dataset Beginners	0	141	January 18, 2024
Allow streaming of large datasets with image/audio 🤗Datasets	18	3980	May 30, 2022
Fetching rows of a large Dataset by index 🤗Datasets	10	1644	March 15, 2021
Why is it so slow to access data through iteration with hugginface dataset? Intermediate	2	2863	July 21, 2022