How to publish a text to-image dataset on huggingface

[namanb]

I essentially want to format my dataset like this- flowers dataset . I am not able to use image dtype with parquet file. Can anyone guide how to do this?

[John6666]

The most reliable way is to load it using the huggingface datasets library and then save it in parquet format. Or rather, I think it’s saved in parquet if you don’t specify anything…

There are various ways to load it, so you can try them out and choose the method that seems to work.

Load Dataset and Save as Parquet

I am curious whether we can directly save a dataset as parquet as in the original dataset at HF. For example fineweb-edu/sample/10BT has 13 parquet files # download the dataset fw = load_dataset( "HuggingFaceFW/fineweb-edu", name="sample-10BT, split="train", num_proc=10, cache_dir=cache_dir, ) fw.to_parquet(parquet_dir) Loads these files as arrow although they are originally parquet, and when saving with to_parquet() it converts arrow to a single parquet file. So my que…

Load image data