Hi,
We tried to use HfApi to get a list of the datasets.
When we run api.list_datasets() - we got exactly 10000 DatasetInfos , although the web count 19,351.
Is it possible that the api uses pagination? If it is so, can you please explain how to use it? If not, can you please explain what are we missing?
Yes, the Hub API now returns a paginated response for the lists. The huggingface_hub library takes care of the pagination and returns the complete list, but you’ll have to update it to 0.11.0 or more recent (Release Extended HfApi, pagination, simplified login and more · huggingface/huggingface_hub · GitHub).
2 Likes