I use data_set = load_dataset(self.data_file_path, cache_dir=cache_dir, split=“train”) Report when loading dataset (approximately 84GB) pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 10761561509， Try setting up according to the help provided in other po…

LoadDataSet pyarrow.lib.ArrowCapacityError

John6666 January 12, 2025, 5:18am 4

How about trying .shard()?

Topic		Replies	Views
Proprietary database load error: TypeError: Argument 'storage' has incorrect type (expected pyarrow.lib.Array, got pyarrow.lib.ChunkedArray) 🤗Datasets	2	1142	January 25, 2022
Exceeded maximum rows when load_dataset for JSON 🤗Datasets	4	1139	April 6, 2023
Arrowmemoryerror: realloc of size 32 GB failed 🤗Datasets	2	3277	January 6, 2023
ArrowNotImplementedError when loading json dataset 🤗Datasets	3	1759	December 17, 2021
Loading dataset from disk taking more time than expected 🤗Datasets	0	717	March 14, 2022