Is it safe to share the cache dir across multiple processes and/or users?
We have a number of users running training job acorss a cluster with several dozen GPUs. We often use the same datasets or datasets derived with transformation of the same base dataset.
In this setting, is ti OK to configure the cache dirs to point to the same location for all of the users?
What happens if one person starts a preprocess that someone else is already running (and would procude the same set of files in the cache)?
Another question (though I think I’d already have seen problems if this wasn’t safe) is what happens if multiple of my own processes try the same transforms (ie want to write to the same cache)?