When first I did
from transformers import BertModel model = BertModel.from_pretrained('bert-base-cased')
Then it’s fine.
But after doing the above, when I do:
from transformers import BertForSequenceClassification m = BertForSequenceClassification.from_pretrained('bert-base-cased')
I get warning messages:
Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias'] - This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']
There is another topic regarding the same issue in the forum here.
What I have understood is that, due to the first code which I ran, the weights of the pre-trained bare
bert-base-cased model got downloaded, and when I ran the second code for sequence classification, the weights regarding the sequence classification didn’t get downloaded because it is grabbing its checkpoint from the first code which I ran.
The same is also given in the last paragraph of the warning message.
So, what’s the solution to download the pre-trained weights for sequence classification tasks or in general other tasks?