Hi,
I’m administrating a small server where people train AI models on. How can I make sure, people don’t download harmful models?
I thought of only allowing the usage of safe tensors or a whitelist of vendors. Is there anything like that?
Best,
Sebastian
Hello, I’m not an expert, but I’d like to help.
If you want to prevent users from training models on harmful data, maybe you could moderate the data using a moderation model. If you want a lighter or lower-cost option, you could also use random sampling and review the sampled data, although that would not guarantee that all harmful data is caught.
I didn’t fully understand your setup, but you mentioned not wanting people to download harmful models, so I assume users may also be downloading models directly. If that is the case, you could restrict downloads by using a whitelist or blocklist of specific accounts, repositories, or model sources. For example, allowing only approved sources may help, because even safetensors models can still be harmful.
If by “harmful” you mean uncensored models, those checks may help. But if you mean models or repositories that could be risky for the server itself, then sandboxing and limiting permissions would also be important.
So the question is really about models containing malware (i.e., data is not a concern).
The setup is, that people ssh onto the machine. They write python code in their repos. Some of them install the huggingface_hub package and download arbitrary models. I read in different news articles that there are many models that contain harmful code that is shipped easily via the pickled model file and is executed as soon as you load the model. I wanted to guard against such a case.
Now you also talked about the safe tensors format and white-/blacklisting of accounts. But I wonder, how this could be done on a technical level. I could imagine to write some kind of config file for hugging face which I could put into any users home directory upon creation or to set some environment variables. So I don’t need to guard against malicous intend of my users, I just want to make sure that they don’t do something stupid accidentally.
Hello! Thanks for completing the design.
You said that you are worried about pickle-based models containing harmful things, and that is true. Pickle files can contain Python objects that may be dangerous when loaded.
I can give you two solutions that I found, though these are only my own suggestions and there may be better ones.
Allow safetensors where possible Since pickle files can contain harmful Python objects, it is better to allow safetensors whenever possible. If needed, you can convert pickle-based models or other formats into safetensors first. This is safer, but for very large models it may be slow and a bit inefficient.
Sandbox every user in a separate virtual machine You can sandbox each user inside a separate virtual machine. Shared drivers and other core components can help speed up VM creation. For each new user, you can create a separate VM with a fresh Python environment, or one that already contains some default libraries if needed. This is usually better for larger models and more general workloads, but it can become harder to maintain as the system grows and may be more complex overall.
So in short, these are the two solutions I found, but there could definitely be better ones depending on the exact setup.
Hi, not an expert but would like to share my suggestion for reference.
Since your goal is to prevent accidental execution rather than stopping determined malicious users, you can handle this locally without needing complex sandboxes.
Network-level filter via HF_ENDPOINT: The huggingface_hub library respects the HF_ENDPOINT environment variable. You can set this globally for all SSH users (e.g., in /etc/profile.d/hf.sh) to point to a lightweight proxy (like a simple Nginx config) running locally on the server. Have the proxy forward all requests to https://huggingface.co, but return a 403 for file extensions associated with pickled code (.bin, .pt, .pth, .pkl). This automatically restricts users to downloading .safetensors and config files.
Python-level guardrail via sitecustomize.py: If you want to catch the execution itself, you can add a sitecustomize.py file to your server’s global Python environment. Because this script executes automatically on Python startup, you can use it to monkey-patch torch.load to raise a custom warning or error. This effectively acts as a tripwire, reminding users to pass use_safetensors=True when loading models via transformers.
Relying on safetensors is exactly the right instinct, you just need to enforce it at the proxy or interpreter level.
VERIFY BEFORE POSTING:
Verify that all download traffic from huggingface_hub strictly routes through HF_ENDPOINT in the version your users are running (it generally does, but it is worth testing in your specific environment).
If you go the sitecustomize.py route, ensure monkey-patching torch.load won’t break users’ legitimate local training workflows (e.g., saving and loading their own optimizer states or mid-training checkpoints, which often default to pickle).