According to datasets document : A few interesting features are provided out-of-the-box by the Apache Arrow backend: - **multi-threaded** or single-threaded reading - automatic decompression of input files (based on the filename extension, such as my_data.csv.gz) - fetching column names …

Read CSV multi threading

imthanhlv July 19, 2021, 11:06am 2

pinging @lhoestq , sorry I can’t find the multi-threaded part in document

Topic		Replies	Views
Local dataset loading performance: HF's arrow vs torch.load 🤗Datasets	5	1191	November 24, 2024
Allow streaming of large datasets with image/audio 🤗Datasets	18	3964	May 30, 2022
Custom 20GB Arrow dataset very slow to train Beginners	1	74	March 20, 2025
Loading dataset from disk taking more time than expected 🤗Datasets	0	714	March 14, 2022
Extremely slow data loading of imagefolder 🤗Datasets	9	2467	January 4, 2024