I use data_set = load_dataset(self.data_file_path, cache_dir=cache_dir, split=“train”) Report when loading dataset (approximately 84GB) pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 10761561509， Try setting up according to the help provided in other po…

LoadDataSet pyarrow.lib.ArrowCapacityError

John6666 January 12, 2025, 5:18am 4

How about trying .shard()?

Topic		Replies	Views
Strange pyarrow error when extracting rows from a public dataset Intermediate	2	42	April 30, 2025
Unable to Load Dataset Using `load_dataset` 🤗Datasets	10	246	March 11, 2025
Proprietary database load error: TypeError: Argument 'storage' has incorrect type (expected pyarrow.lib.Array, got pyarrow.lib.ChunkedArray) 🤗Datasets	2	1132	January 25, 2022
Exceeded maximum rows when load_dataset for JSON 🤗Datasets	4	1138	April 6, 2023
Arrowmemoryerror: realloc of size 32 GB failed 🤗Datasets	2	3240	January 6, 2023