Understanding where model weights are stored for research project on AI openness

Hello there,

I have a question regarding how model weights are generally stored on Hugging Face. I am conducting research on the openness of AI models and understanding when model weights are published and what effects this might have is crucial. One aspect of my project therefore involves exploring the different file formats in which model weights can be stored (e.g., .pt files, .bin files, or other formats).

If anyone has pointers or resources, I would greatly appreciate it. I realize this is a broad set of questions, so thank you in advance for your and guidance.

Thank you again, and apologies for my ignorance regarding this matter.

Best,
Fabian

1 Like

Hello. Since it’s a broad topic, let’s start by talking about the format. There are no particular restrictions on the file format of the weights of the models uploaded to Hugging Face. People are doing what they want, from Python pickles like you mentioned to quantized GGUF files, etc.

However, the safetensors file format is usually recommended by Hugging Face, and it has been widely used for the past few years. It is similar to Python’s pickle in terms of how it is used, but it is a file format that eliminates security risks as much as possible. You can think of it as containing only metadata and a state_dict.
Because this file format is recommended, it also has an advantage when searching in HF, for example, you can get metadata with relatively low overhead.

3 Likes

Thank you for your response. This helps me a lot!

1 Like

In addition to @John6666 response, the Python Pickles format has security vulnerability because malicious models can be uploaded using that format and it may remain undetectable on hf, also ggufs are mostly quantized version of the original safetensor model.

2 Likes