I converted ImageNet and its corresponding depth images into Parquet format using save_to_disk, storing them as a DatasetDict object. I can successfully load the dataset using load_from_disk as follows: from datasets import load_from_disk ds = load_from_disk("/defaultShare/pubdata/ImageNet_arrow_r…

max_shard_size may be too large. PyArrow Dataset error when calling `load_dataset` opened 01:16AM - 20 Jul 22 UTC (UTC) [image] piraka9011 bug …

Unable to Load Dataset Using `load_dataset`

system Closed March 11, 2025, 12:47am 11

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Custom 20GB Arrow dataset very slow to train Beginners	1	78	March 20, 2025
LoadDataSet pyarrow.lib.ArrowCapacityError 🤗Datasets	6	319	January 12, 2025
Strange pyarrow error when extracting rows from a public dataset Intermediate	2	82	April 30, 2025
Cannot load dataset on Kaggle 🤗Datasets	4	3145	August 16, 2023
Strange Error While Attempting to Load DataSet 🤗Datasets	7	3632	March 28, 2025