Hi, I am performing pre-training using an ElectraMaskedLM model. Initially I ran the pre-training for 10 epochs. After that I wanted to continue the pre-training for 20 more epochs. But while loading the model I find a warning that says, Some weights of the model checkpoint at ./output/checkpoint-2766450/ were not used when initializing ElectraForMaskedLM: and in the layers list it consists all the layers of my model. I am aware of that if I change my model from MaskedLM to SequenceClassification or any downstream task models, the warning comes for the last few set of layers due to the change in architecture, but in my case there is no change in the model and the warning comes for all the layers. The full warning message is below,
Some weights of the model checkpoint at ./output/checkpoint-2766450/ were not used when initializing ElectraForMaskedLM: ['module.electra.encoder.layer.9.attention.self.key.weight', 'module.electra.encoder.layer.4.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.5.attention.self.value.weight', 'module.electra.encoder.layer.8.attention.self.query.bias', 'module.electra.encoder.layer.5.attention.output.dense.weight', 'module.electra.encoder.layer.0.attention.output.dense.bias', 'module.electra.encoder.layer.8.output.dense.weight', 'module.electra.encoder.layer.11.attention.self.query.bias', 'module.electra.encoder.layer.3.attention.self.value.weight', 'module.electra.encoder.layer.6.attention.self.query.weight', 'module.electra.encoder.layer.9.intermediate.dense.bias', 'module.electra.encoder.layer.6.attention.output.dense.bias', 'module.electra.encoder.layer.2.output.LayerNorm.bias', 'module.electra.encoder.layer.8.output.dense.bias', 'module.electra.encoder.layer.8.output.LayerNorm.bias', 'module.electra.encoder.layer.7.attention.output.dense.weight', 'module.electra.encoder.layer.0.attention.self.key.bias', 'module.electra.encoder.layer.1.attention.output.dense.weight', 'module.electra.encoder.layer.10.attention.output.dense.weight', 'module.electra.encoder.layer.11.intermediate.dense.weight', 'module.electra.embeddings.LayerNorm.weight', 'module.electra.encoder.layer.6.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.11.output.dense.weight', 'module.electra.encoder.layer.3.attention.output.dense.bias', 'module.electra.encoder.layer.8.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.9.attention.self.query.bias', 'module.electra.encoder.layer.5.output.dense.bias', 'module.electra.encoder.layer.9.output.LayerNorm.weight', 'module.electra.encoder.layer.4.attention.self.query.weight', 'module.generator_predictions.LayerNorm.bias', 'module.electra.encoder.layer.10.intermediate.dense.weight', 'module.electra.encoder.layer.1.output.dense.bias', 'module.electra.encoder.layer.8.attention.output.dense.weight', 'module.generator_lm_head.bias', 'module.electra.encoder.layer.3.intermediate.dense.bias', 'module.electra.encoder.layer.9.attention.output.dense.weight', 'module.electra.encoder.layer.3.attention.self.key.weight', 'module.electra.encoder.layer.1.output.LayerNorm.weight', 'module.electra.encoder.layer.2.attention.output.dense.weight', 'module.electra.encoder.layer.4.attention.output.dense.bias', 'module.electra.encoder.layer.7.attention.self.query.weight', 'module.electra.encoder.layer.5.attention.output.dense.bias', 'module.electra.encoder.layer.7.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.8.output.LayerNorm.weight', 'module.electra.encoder.layer.8.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.0.output.dense.bias', 'module.electra.encoder.layer.0.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.6.attention.output.LayerNorm.bias', 'module.generator_predictions.LayerNorm.weight', 'module.electra.encoder.layer.2.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.6.intermediate.dense.bias', 'module.electra.embeddings.word_embeddings.weight', 'module.electra.encoder.layer.1.intermediate.dense.weight', 'module.electra.encoder.layer.1.attention.self.key.bias', 'module.electra.encoder.layer.3.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.10.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.11.attention.self.value.bias', 'module.electra.encoder.layer.0.output.LayerNorm.weight', 'module.electra.encoder.layer.5.attention.self.key.weight', 'module.generator_predictions.dense.bias', 'module.electra.encoder.layer.10.attention.self.key.bias', 'module.electra.encoder.layer.2.attention.self.key.weight', 'module.electra.encoder.layer.9.attention.self.key.bias', 'module.electra.encoder.layer.7.attention.self.key.bias', 'module.electra.encoder.layer.0.attention.self.value.weight', 'module.electra.encoder.layer.0.attention.self.query.bias', 'module.electra.encoder.layer.8.attention.self.value.bias', 'module.electra.encoder.layer.4.attention.self.key.weight', 'module.electra.encoder.layer.6.attention.self.value.weight', 'module.electra.encoder.layer.11.output.LayerNorm.bias', 'module.electra.encoder.layer.2.output.LayerNorm.weight', 'module.electra.encoder.layer.6.attention.output.dense.weight', 'module.electra.encoder.layer.4.output.dense.bias', 'module.electra.encoder.layer.3.intermediate.dense.weight', 'module.electra.encoder.layer.7.output.LayerNorm.bias', 'module.electra.encoder.layer.1.attention.self.value.bias', 'module.electra.embeddings.position_embeddings.weight', 'module.electra.encoder.layer.11.attention.self.query.weight', 'module.electra.encoder.layer.4.attention.self.key.bias', 'module.electra.encoder.layer.4.attention.self.value.weight', 'module.generator_predictions.dense.weight', 'module.electra.encoder.layer.1.output.dense.weight', 'module.electra.encoder.layer.10.attention.self.key.weight', 'module.electra.encoder.layer.11.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.6.attention.self.query.bias', 'module.electra.encoder.layer.0.output.LayerNorm.bias', 'module.electra.encoder.layer.0.intermediate.dense.bias', 'module.electra.encoder.layer.4.attention.self.value.bias', 'module.electra.encoder.layer.6.intermediate.dense.weight', 'module.electra.encoder.layer.9.attention.self.value.weight', 'module.electra.encoder.layer.10.attention.self.value.weight', 'module.electra.encoder.layer.10.output.dense.bias', 'module.electra.encoder.layer.6.output.LayerNorm.weight', 'module.electra.encoder.layer.0.output.dense.weight', 'module.electra.encoder.layer.1.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.10.attention.self.query.bias', 'module.electra.encoder.layer.10.intermediate.dense.bias', 'module.electra.encoder.layer.1.attention.self.query.bias', 'module.electra.encoder.layer.7.intermediate.dense.bias', 'module.electra.encoder.layer.2.attention.self.value.bias', 'module.electra.encoder.layer.9.attention.self.query.weight', 'module.electra.encoder.layer.1.intermediate.dense.bias', 'module.generator_lm_head.weight', 'module.electra.encoder.layer.1.attention.self.value.weight', 'module.electra.encoder.layer.3.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.9.output.dense.weight', 'module.electra.encoder.layer.10.output.dense.weight', 'module.electra.encoder.layer.5.intermediate.dense.weight', 'module.electra.encoder.layer.4.intermediate.dense.bias', 'module.electra.encoder.layer.9.output.dense.bias', 'module.electra.encoder.layer.1.output.LayerNorm.bias', 'module.electra.encoder.layer.2.output.dense.bias', 'module.electra.encoder.layer.2.intermediate.dense.weight', 'module.electra.encoder.layer.5.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.8.attention.self.query.weight', 'module.electra.encoder.layer.7.attention.self.query.bias', 'module.electra.encoder.layer.2.attention.self.value.weight', 'module.electra.encoder.layer.7.output.LayerNorm.weight', 'module.electra.encoder.layer.10.output.LayerNorm.weight', 'module.electra.encoder.layer.3.attention.self.value.bias', 'module.electra.encoder.layer.11.output.dense.bias', 'module.electra.encoder.layer.0.attention.self.key.weight', 'module.electra.encoder.layer.3.attention.self.query.weight', 'module.electra.encoder.layer.11.attention.output.dense.weight', 'module.electra.encoder.layer.7.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.3.output.dense.weight', 'module.electra.encoder.layer.4.attention.output.dense.weight', 'module.electra.encoder.layer.8.intermediate.dense.weight', 'module.electra.encoder.layer.1.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.6.output.LayerNorm.bias', 'module.electra.encoder.layer.4.attention.output.LayerNorm.weight', 'module.electra.embeddings.LayerNorm.bias', 'module.electra.encoder.layer.2.intermediate.dense.bias', 'module.electra.encoder.layer.2.output.dense.weight', 'module.electra.encoder.layer.11.output.LayerNorm.weight', 'module.electra.encoder.layer.0.attention.output.dense.weight', 'module.electra.encoder.layer.9.attention.self.value.bias', 'module.electra.encoder.layer.0.attention.self.query.weight', 'module.electra.encoder.layer.5.attention.self.query.bias', 'module.electra.encoder.layer.5.attention.self.value.bias', 'module.electra.encoder.layer.5.output.dense.weight', 'module.electra.encoder.layer.3.attention.self.key.bias', 'module.electra.encoder.layer.6.attention.self.value.bias', 'module.electra.encoder.layer.6.attention.self.key.bias', 'module.electra.encoder.layer.2.attention.output.dense.bias', 'module.electra.embeddings.position_ids', 'module.electra.encoder.layer.3.output.LayerNorm.weight', 'module.electra.encoder.layer.7.attention.self.value.weight', 'module.electra.encoder.layer.11.attention.output.dense.bias', 'module.electra.encoder.layer.10.attention.output.dense.bias', 'module.electra.encoder.layer.4.output.LayerNorm.bias', 'module.electra.encoder.layer.7.attention.self.key.weight', 'module.electra.encoder.layer.11.attention.self.value.weight', 'module.electra.encoder.layer.7.attention.output.dense.bias', 'module.electra.embeddings.token_type_embeddings.weight', 'module.electra.encoder.layer.8.attention.output.dense.bias', 'module.electra.encoder.layer.5.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.5.attention.self.key.bias', 'module.electra.encoder.layer.2.attention.self.query.weight', 'module.electra.encoder.layer.8.attention.self.key.bias', 'module.electra.encoder.layer.9.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.1.attention.self.key.weight', 'module.electra.encoder.layer.3.output.dense.bias', 'module.electra.encoder.layer.0.attention.self.value.bias', 'module.electra.encoder.layer.11.attention.self.key.bias', 'module.electra.encoder.layer.4.output.LayerNorm.weight', 'module.electra.encoder.layer.3.attention.output.dense.weight', 'module.electra.encoder.layer.0.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.6.attention.self.key.weight', 'module.electra.encoder.layer.10.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.9.output.LayerNorm.bias', 'module.electra.encoder.layer.5.intermediate.dense.bias', 'module.electra.encoder.layer.11.attention.self.key.weight', 'module.electra.encoder.layer.9.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.2.attention.output.LayerNorm.weight', 'module.electra.encoder.layer.5.output.LayerNorm.weight', 'module.electra.encoder.layer.10.output.LayerNorm.bias', 'module.electra.encoder.layer.4.intermediate.dense.weight', 'module.electra.encoder.layer.7.output.dense.weight', 'module.electra.encoder.layer.2.attention.self.key.bias', 'module.electra.encoder.layer.2.attention.self.query.bias', 'module.electra.encoder.layer.8.intermediate.dense.bias', 'module.electra.encoder.layer.11.intermediate.dense.bias', 'module.electra.encoder.layer.3.attention.self.query.bias', 'module.electra.encoder.layer.10.attention.self.value.bias', 'module.electra.encoder.layer.10.attention.self.query.weight', 'module.electra.encoder.layer.6.output.dense.weight', 'module.electra.encoder.layer.7.output.dense.bias', 'module.electra.encoder.layer.0.intermediate.dense.weight', 'module.electra.encoder.layer.7.attention.self.value.bias', 'module.electra.encoder.layer.1.attention.output.dense.bias', 'module.electra.encoder.layer.8.attention.self.value.weight', 'module.electra.encoder.layer.11.attention.output.LayerNorm.bias', 'module.electra.encoder.layer.5.attention.self.query.weight', 'module.electra.encoder.layer.4.attention.self.query.bias', 'module.electra.encoder.layer.4.output.dense.weight', 'module.electra.encoder.layer.8.attention.self.key.weight', 'module.electra.encoder.layer.3.output.LayerNorm.bias', 'module.electra.encoder.layer.1.attention.self.query.weight', 'module.electra.encoder.layer.7.intermediate.dense.weight', 'module.electra.encoder.layer.9.attention.output.dense.bias', 'module.electra.encoder.layer.9.intermediate.dense.weight', 'module.electra.encoder.layer.6.output.dense.bias', 'module.electra.encoder.layer.5.output.LayerNorm.bias']
- This IS expected if you are initializing ElectraForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ElectraForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of ElectraForMaskedLM were not initialized from the model checkpoint at ./output/checkpoint-2766450/ and are newly initialized: ['encoder.layer.4.attention.self.value.bias', 'embeddings.word_embeddings.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.output.LayerNorm.bias', 'embeddings.position_embeddings.weight', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.2.attention.self.query.bias', 'embeddings.token_type_embeddings.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.8.output.dense.bias', 'encoder.layer.5.output.dense.bias', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'embeddings_project.bias', 'encoder.layer.0.output.dense.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.1.output.dense.weight', 'encoder.layer.11.output.LayerNorm.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.4.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.6.attention.self.key.weight', 'generator_predictions.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.2.output.dense.weight', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.3.output.dense.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.2.attention.self.value.bias', 'generator_lm_head.weight', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.6.output.dense.bias', 'generator_predictions.dense.bias', 'encoder.layer.10.attention.self.value.bias', 'embeddings_project.weight', 'encoder.layer.11.output.dense.weight', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.3.output.dense.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.9.attention.output.dense.bias', 'generator_predictions.LayerNorm.bias', 'generator_predictions.dense.weight', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.7.attention.self.query.weight', 'embeddings.LayerNorm.bias', 'encoder.layer.0.intermediate.dense.bias', 'embeddings.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.10.output.LayerNorm.weight', 'generator_lm_head.bias', 'encoder.layer.10.output.dense.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.9.output.dense.weight', 'encoder.layer.0.attention.self.query.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.