I use data_set = load_dataset(self.data_file_path, cache_dir=cache_dir, split=“train”) Report when loading dataset (approximately 84GB) pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 10761561509， Try setting up according to the help provided in other po…

LoadDataSet pyarrow.lib.ArrowCapacityError

Dammond January 12, 2025, 5:45am 5

Thank you for your reply.
Is. shard() partitioned only when generating the load_dataset() object

But this error occurred during load_dataset()

1 Like

Topic		Replies	Views
Strange pyarrow error when extracting rows from a public dataset Intermediate	2	212	April 30, 2025
Unable to Load Dataset Using `load_dataset` 🤗Datasets	10	682	March 11, 2025
Proprietary database load error: TypeError: Argument 'storage' has incorrect type (expected pyarrow.lib.Array, got pyarrow.lib.ChunkedArray) 🤗Datasets	2	1161	January 25, 2022
Error converting np float32 🤗Datasets	3	45	August 5, 2025
Exceeded maximum rows when load_dataset for JSON 🤗Datasets	4	1153	April 6, 2023