Hello,
I’m having some trouble with getting the huggingface cache to work for my particular use case. I’ve been setting up kubernetes training for my transformer models, and the way it works is that pods are initialized from a docker image and then I mount a persistent disk with the data. Since pods are initialized from a docker image, any other files relevant for training (e.g. the training data) must be stored on the persistent disk. This disk is mounted in readonly mode due to certain limitations outside of my control.
Currently, downloading the model checkpoint for whichever architecture I wish to fine-tune eats up a lot of time, so I’d like to take advantage of the HF cache. The problem is that if I do so by downloading the models to a cache folder on the previously mentioned persistent disk, then setting the HF_HOME environment variable to point to that folder, my training runs will crash since huggingface tries to write to the HF_HOME folder for some reason. The model weights are already fully downloaded and have not been updated since, so I’m not really sure why this is the case. I realize that when running AutoModel.from_pretrained I could provide a local filepath instead of the pretrained model name, but then I’m basically managing the cache myself and I’m wondering if instead there is a setting for this.
In short, how can I use cached, already downloaded model weights from a readonly filesystem?