Clarification on Dataset Size Discrepancy – Common Pile v0.1

🤗Datasets
1

Clarification regarding the Common Pile v0.1 dataset, specifically the discrepancy between the size mentioned in the official documentation and the total size of the data available for download via the Hugging Face Hub. According to the arXiv paper (2506.05209v1) and the website, the full Common Pile v0.1 dataset is approximately 8 TB in size. However, after downloading all available .json.gz and .jsonl.gz files from the common-pile/* repositories, the total size amounts to approximately 3.6 TB.

Could you kindly clarify the following:

  1. Does the 8 TB include those not exposed via the UI or HfFileSystem?
  2. Are there any dataset components stored in separate repositories or requiring additional authorization or private access?
  3. Is the discrepancy due to compression?
1 Like
2

3

Possible.

2

If it’s gated, I think you can see the size. If it’s private, no one else can see it, including me. If it’s a dataset repository that uses a dataset builder script, I think there may be some with small sizes, but from what I’ve seen, most seem to be repositories that just store the data itself.

1

I think it would be quicker to ask the author.