I use data_set = load_dataset(self.data_file_path, cache_dir=cache_dir, split=“train”) Report when loading dataset (approximately 84GB) pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 10761561509， Try setting up according to the help provided in other po…

LoadDataSet pyarrow.lib.ArrowCapacityError

John6666 January 12, 2025, 5:18am 4

How about trying .shard()?

Topic		Replies	Views
Error While Saving Dataset with PyArrow 🤗Datasets	0	42	December 14, 2024
Dataset too large error 🤗Datasets	1	788	March 15, 2023
Arrowmemoryerror: realloc of size 32 GB failed 🤗Datasets	2	3070	January 6, 2023
Problem loading datasets library from Kaggle 🤗Datasets	6	5681	October 12, 2021
Proprietary database load error: TypeError: Argument 'storage' has incorrect type (expected pyarrow.lib.Array, got pyarrow.lib.ChunkedArray) 🤗Datasets	2	1106	January 25, 2022