Dataset too large error

🤗Datasets
1

image
image1894Ă—1032 244 KB

If my images dataset more than 20000, always give me this error.
ArrowInvalid: JSON parse error: Column() changed from object to string in row 0
Can somebody help me?QAQ

2

Hi! I think the problem is the format of the metadata file rather than the dataset size. You can test this with the following code:

import pyarrow.json as paj
table = paj.read_json("path/to/metadata_file")

You can find the correct metadata format here.

1 Like