Custom loading dataset script

🤗Datasets
1

I have prepared and tested a custom loading dataset script. Although the test was successful, it hasn’t created the metadata file dataset_info.json in my dataset folder. Instead it has created a README.md file. What am I missing here?

Output from the test

datasets-cli test datasets/tse --save_infos --all_configs
Testing builder 'entity' (1/2)
Downloading and preparing dataset tse/entity to /Users/home/.cache/huggingface/datasets/tse/entity/1.0.0/7f33fefc7622e65f7e037dda1222c13ada1717400eef6ed51970adf336261e21...
Dataset tse downloaded and prepared to /Users/home/.cache/huggingface/datasets/tse/entity/1.0.0/7f33fefc7622e65f7e037dda1222c13ada1717400eef6ed51970adf336261e21. Subsequent calls will reuse this data.                                                      
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 321.33it/s]
Dataset card saved at datasets/tse/README.md
Testing builder 'queries' (2/2)
Downloading and preparing dataset tse/queries to /Users/home/.cache/huggingface/datasets/tse/queries/1.0.0/7f33fefc7622e65f7e037dda1222c13ada1717400eef6ed51970adf336261e21...
Dataset tse downloaded and prepared to /Users/home/.cache/huggingface/datasets/tse/queries/1.0.0/7f33fefc7622e65f7e037dda1222c13ada1717400eef6ed51970adf336261e21. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 25.49it/s]
Dataset card saved at datasets/tse/README.md
Test successful.

Directory listing

(base) home@sls-MacBook-Pro tse % ls -l
total 32
-rw-r--r--  1 home  staff  1311 Dec 25 16:18 README.md
drwxr-xr-x  4 home  staff   128 Dec 25 14:20 data
-rw-r--r--  1 home  staff  9500 Dec 25 16:16 tse.py

For the final test, why do we need to clone the huggingface/datasets repo to run the following command
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_tse

Could you please explain?

2

I was looking at the older version of the instructions, here dataset script

Just to be certain, we don’t require a dataset_info.json file. Is this correct?

3

Got another query.

How do we pass additional arguments to the _generate_examples() method from the load_dataset() function?

5

I figured out how to pass additional arguments to the function. Thanks!

If you could answer my earlier question, I would appreciate it.

6

Hi ! We now store the dataset infos into the README.md yaml tags instead of using a JSON file, in order to have everything in one place :slight_smile: