If the library is writing to the cache itself, there’s nothing you can do…
If the library is only writing to HF_HOME for other processes than caching, you might be able to get around it by using HF_HUB_CACHE instead of HF_HOME. Because HF_HOME has a wide range of effects…