I am doing the following three steps for a large number of iterations:
Loading a parquet file using load_dataset().
Tokenize it using dataset.map() and HuggingFace tokenizers.
Saving the tokenised dataset on disk in arrow format.
I have the following questions:
a) I don’t want to save anything to the ~/.cache/huggingface/datasets/ as I am saving the final result at a separate location for further use. I tried using load_dataset(..., cache_dir=None) and setting datasets.disable_caching() but none seem to work. From some other threads, I understood that caching can be disabled in dataset.map() and dataset.filter() but not in load_dataset(). How do I disable all types of caching?
b) I plan to train a GPT like transformer model on this tokenised data using the HF ecosystem. I want to conserve disk space but at the same time not make loading extremely slow downstream. What is better arrow vs parquet format in Step 3 above?
a) I don’t want to save anything to the ~/.cache/huggingface/datasets/ as I am saving the final result at a separate location for further use. I tried using load_dataset(..., cache_dir=None) and setting datasets.disable_caching() but none seem to work. From some other threads, I understood that caching can be disabled in dataset.map() and dataset.filter() but not in load_dataset() . How do I disable all types of caching?
Indeed currently disable_caching uses a temp directory when saving intermediate map results, but load_dataset still writes the original dataset in ~/.cache/huggingface/datasets/
b) I plan to train a GPT like transformer model on this tokenised data using the HF ecosystem. I want to conserve disk space but at the same time not make loading extremely slow downstream. What is better arrow vs parquet format in Step 3 above?
It depends on the dataset size and your training setup, but usually it’s fine using Arrow. For bigger datasets you may use Parquet instead and use streaming=True
I agree it’s kinda confusing though, we might change that at one point and make load_dataset use a temporary directory in that case, though it’s maybe not a trivial change
I am getting very different performance results, when running to consecutive runs with the same code (first run: 99% accuracy, second run: 60%). I am using the datasets package and dataset cifar10. There is no error message or anything. After two months of trying to pin down the bug, I am guessing it is a caching problem. Is there a way to visualize the caching structure of datasets?
I also noticed that every function of datasets has a caching dir, which make it really hard for me to get a sense of how the caching structure is.
Do you have any advice how to make runs reproducible?