"Some weights were not used" message with AutoModel

Beginners
1

Hi, I’m a newbie trying to learn Hugginface transformers api.
The line

model = AutoModel.from_pretrained(“bert-base-cased”)

give the following message:

Some weights of the model checkpoint at bert-base-cased were not used when initializing BertModel: [‘cls.predictions.transform.dense.bias’, ‘cls.predictions.bias’, ‘cls.seq_relationship.weight’, ‘cls.predictions.transform.LayerNorm.bias’, ‘cls.seq_relationship.bias’, ‘cls.predictions.transform.LayerNorm.weight’, ‘cls.predictions.decoder.weight’, ‘cls.predictions.transform.dense.weight’]

I didn’t expect that. I know you can obtain such a message if you init a ModelForSomething using a checkpoint for a basic model, but this is not the case.

Any help?
Thanks

2

Probably I understood the problem: in the checkpoint there are some weights related to training layers that are dropped and are not present in BertModel class. So it shouln’t be a problem

3

Yup, exactly!

4

Thank you for replying… When I read more carefully the warning I realized that there were not uninitialized weights in the model (it can happen if you use a base model checkpoint for initializing a ModelForSomething) but the opposite: there were unused checkpoint weights