No Biases for Llama-3.2-3B-Instruct

Models
1

I used this code to extract the weights and biases of the model:

from transformers import AutoModelForCausalLM

model_name = "meta-llama/Llama-3.2-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name)
model_weights = model.state_dict()
weights = {}
biases = {}

for key, value in model_weights.items():
    if 'weight' in key:
        weights[key] = value
    else:
        biases[key] = value

print("Weights:", weights)
print("Biases:", biases)

However, the output renders the biases dictionary empty.

Is this intentional behaviour that this model do not have any biases?

Someone please explain this, since biases is one of the fundemental parts of NNs and can significantly boost preformance. Why did meta do this?

P.S. Just to be sure, I added this check for the length of weights:

print(len(weights.keys()), len(model_weights.keys()))

and the output:

255 255

Meaning there REALLY is not bias in this model (is it what Meta mean by models not having biases?)

1 Like