Writing custom dataset script with files residing in local

🤗Datasets
1

requirement: I have downloaded zipped file < training_set → images & annotations +test_set → images and annotations. I want to create custom dataset generator which will be consumed by model and run inference.
I am using this url as reference - Create a dataset loading script and trying to configure the load_datasets into parsing the files and loading them in for model inference.
I tried :
dataset_config[“LOADING_SCRIPT_FILES”],
dataset_config[“CONFIG_NAME”],
data_dir=dataset_config[“DATA_DIR”],
cache_dir=dataset_config[“CACHE_DIR”]

With each of the location being pointed to my local directory where my data resides. The extract function within the builder module is taking long time and it is always downloading the dataset from external sources rather taking it from the unzipped folder location of the data in my local. Please suggest a solution or a started code I can refer to

2

Instead of the dataset script approach, you can use Dataset.from_generator and pass a generator that combines images with annotations to avoid caching extracted data files.

1 Like