How to handle the cache system properly?

[jtourille]

Hello,

I’ve been struggling with large datasets on our HPC cluster and I think I don’t understand fully how to use the datasets library.

I want to use fineweb-edu, so I downloaded the dataset using snapshot_download as presented below. I skip the sample parts. The dataset is stored on a NFS server.

...

_ = snapshot_download(
    "HuggingFaceFW/fineweb-edu",
    repo_type="dataset",
    allow_patterns=["data/*"],
    local_dir=os.path.join(target_dir, "HuggingFaceFW/fineweb-edu"),
)

...

Now I want to load this dataset. As I understand, load_dataset will create an optimized cache version of this dataset. So I specify the location of the cache using the HF_HOME variable and load the dataset.

...

_ = load_dataset(os.path.join(target_dir, "HuggingFaceFW/fineweb-edu", "data"), num_proc=12)

...

The cached version is correctly built, so far so good. Now I want to load this cached version without having access to the original parquet files. How can I do that ? I’ve read the documentation several times and the best way seems to use the arrow builder with load_dataset("arrow", data_files=data_files) but the process is building another cached version of this dataset. I’ve tried to disable caching without success.

I am sure I am missing something, could someone point me in the right direction ?

Thanks,

Julien

[John6666]

Some resources for now:

Cache management

Best way to access the cached transformation arrow file

I was experimenting with the cached arrow dataset but ran into an error The steps followed Loaded the dataset from HF using load_dataset() I created a simple transformation by shuffling the data dataset = dataset["test"].shuffle(seed=42) This step produced a new cache file in the hugging face cache directory - ~/.cache/huggingface/datasets/bt-tech___ofac/dataset/1.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559/cache-c7f4daa392c5ed6a.arrow After I restart the sessio…

Load dataset from a specific cache file

Dear community, Does anyone know if there is a way to load dataset from a specific cached file? I applied very costly transformations using a map function to do some data augmentation and finally, I had my dataset. However, now in my main script, I would like to use the cached file built from another script. Is there any way to call a specific cache file from ~/.cache/huggingface/datasets/ ?

[jtourille]

Thanks for the pointers, I’ve read them already.

I think my question could be rephrased: how can we manage large datasets in large companies ? More precisely, I am looking at a way to avoid to have multiple copies of the same dataset at different locations and minimize the number of download (ideally only one).

I’ve been experimenting since my initial question and I’ve come up with this workflow:

1. Download the dataset with snapshot_download() or git.
2. One person load the dataset with load_dataset() and export it with save_to_disk()
3. Other people that want to use the dataset can make a local copy, do their stuff and remove everything afterwards.

Does that make sense ?

One thing I don’t understand is that load_dataset() and save_to_disk() both save the dataset in arrow format. However, as far as I understand, they do not perform the same optimizations, and therefore load_from_disk cannot load the dataset directly from the cache. Is there any particular reason for that ?

Bests

Julien

[John6666]

load_dataset() and save_to_disk()

Even though they are saved in the same Arrow format, there is a difference in purpose between long-term storage and internal caching for speed optimization. Some people seem to be trying to reuse them.

Additionally, while I am unsure how robust it is in multi-process or multi-user environments, one possible solution to avoid re-downloading the cache is to simply set HF_HOME to a relatively fast shared folder on the network. For remote environments, services like S3 seem to be available by default. However, this does come at a cost…

By the way, for enterprise use cases, there is an option to consult dedicated support on Hugging Face. Whether this is suitable or not will depend on the scale of the project.