I would like to fine-tune a pre-trained transformers model on Question Answering. The model was pre-trained on large engineering & science related corpora.
I have been provided a “checkpoint.pt” file containing the weights of the model. They have also provided me with a “bert_config.json” file but I am not sure if this is the correct configuration file.
from transformers import AutoModel, AutoTokenizer, AutoConfig MODEL_PATH = "./checkpoint.pt" config = AutoConfig.from_pretrained("./bert_config.json") model = AutoModel.from_pretrained(MODEL_PATH, config=config)
The reason I believe that bert_config.json doesn’t match “./checkpoint.pt” file is that, when I load the model with the code above, I get the error that goes as below.
Some weights of the model checkpoint at ./aerobert/phase2_ckpt_4302592.pt were not used when initializing BertModel: [‘files’, ‘optimizer’, ‘model’, ‘master params’]
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertModel were not initialized from the model checkpoint at ./aerobert/phase2_ckpt_4302592.pt and are newly initialized: [‘encoder.layer.2.attention.output.LayerNorm.weight’, ‘encoder.layer.6.output.LayerNorm.bias’, ‘encoder.layer.7.intermediate.dense.bias’, ‘encoder.layer.2.output.LayerNorm.bias’, ‘encoder.layer.21.attention.self.value.bias’, ‘encoder.layer.11.attention.self.value.bias’, …
If I am correct in assuming that “bert_config.json” is not the correct one, is there a way to load this model correctly without the config.json file?