Updating existing message as I am not allowed by HF to post more messages on single thread as a newbie. Anyhow, I resolved the error by using “pip install” instead of “conda install” of datasets. Seems like the conda datasets packages are not updated.
And this is the exact error stack.
$ python .//project_gutenberg_tests.py
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52/52 [00:00<00:00, 291.93it/s]
Traceback (most recent call last):
File "./project_gutenberg_tests.py", line 4, in <module>
dataset = load_dataset("manu/project_gutenberg")
File "envname/lib/python3.10/site-packages/datasets/load.py", line 2108, in load_dataset
ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
File "envname/lib/python3.10/site-packages/datasets/builder.py", line 1125, in as_dataset
datasets = map_nested(
File "envname/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 511, in map_nested
mapped = [
File "envname/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 512, in <listcomp>
_single_map_nested((function, obj, batched, batch_size, types, None, True, None))
File "envname/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 373, in _single_map_nested
return function(data_struct)
File "envname/lib/python3.10/site-packages/datasets/builder.py", line 1155, in _build_single_dataset
ds = self._as_dataset(
File "envname/lib/python3.10/site-packages/datasets/builder.py", line 1229, in _as_dataset
dataset_kwargs = ArrowReader(cache_dir, self.info).read(
File "envname/lib/python3.10/site-packages/datasets/arrow_reader.py", line 252, in read
return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
File "envname/lib/python3.10/site-packages/datasets/arrow_reader.py", line 273, in read_files
pa_table = self._read_files(files, in_memory=in_memory)
File "envname/lib/python3.10/site-packages/datasets/arrow_reader.py", line 216, in _read_files
pa_table = concat_tables(pa_tables) if len(pa_tables) != 1 else pa_tables[0]
File "envname/lib/python3.10/site-packages/datasets/table.py", line 1766, in concat_tables
return ConcatenationTable.from_tables(tables, axis=axis)
File "envname/lib/python3.10/site-packages/datasets/table.py", line 1472, in from_tables
return cls.from_blocks(blocks)
File "envname/lib/python3.10/site-packages/datasets/table.py", line 1385, in from_blocks
table = cls._concat_blocks(blocks, axis=0)
File "envname/lib/python3.10/site-packages/datasets/table.py", line 1331, in _concat_blocks
return pa.concat_tables(pa_tables, promote_options="default")
File "pyarrow/table.pxi", line 5165, in pyarrow.lib.concat_tables
TypeError: concat_tables() got an unexpected keyword argument 'promote_options'```