Dataset generation error after downloading all the parquet files

🤗Datasets
1

Tried both datasets the-stack-dedup and the-stack .
After spending past 5 days downloading and extracting I get a bug:

Extracting data files: 100%|███████████████| 1/1 [03:32<00:00, 212.39s/it]
Traceback (most recent call last):
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 1858, in _prepare_split_single
for _, table in generator:
File “/media/env/lib/python3.11/site-packages/datasets/packaged_modules/parquet/parquet.py”, line 67, in _generate_tables
parquet_file = pq.ParquetFile(f)
^^^^^^^^^^^^^^^^^
File “/media/env/lib/python3.11/site-packages/pyarrow/parquet/core.py”, line 334, in init
self.reader.open(
File “pyarrow/_parquet.pyx”, line 1220, in pyarrow._parquet.ParquetReader.open
File “pyarrow/error.pxi”, line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “”, line 1, in
File “/media/env/lib/python3.11/site-packages/datasets/load.py”, line 1797, in load_dataset
builder_instance.download_and_prepare(
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 890, in download_and_prepare
self._download_and_prepare(
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 985, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 1746, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 1891, in _prepare_split_single
raise DatasetGenerationError(“An error occurred while generating the dataset”) from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

2

This is the error:

pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

So the files you are passing to load_dataset are not Parquet files.

3

I’m getting the same error when downloading the OpenOrca dataset from HuggingFace. These aren’t my parquet files, they are on huggingface.

Was this ever resolved?

4

I’m not able to reproduce the error. Most likely, one of the downloaded Parquet files is corrupted in your case, so pass downlaod_mode="force_redownload" to load_dataset to re-download them.

5

Thanks for the response. The problem was the datasets version. I was using v2.13 and when I upgraded to the latest the problem went away. Open-Orca/Mistral-7B-OpenOrca · OpenOrca Dataset 'Fail to generate dataset'

1 Like
6

Thanks! This also solved my problem.

7

I see the same error with dataset from tatsu-lab/alpaca at main.
I don’t suspect Parquet format changed in a breaking fashion. I’m using datasets in Python (3.2.0) which is current for reading Parquet files.

I’m working on a “getting started” tutorial for experienced Python programmers wanting to build an LLM model from scratch, and picked Alpaca. Perhaps there is a better choice?

1 Like