Is it possible to use a pre-trained Bert model with a modified type_vocab_size parameter?

Beginners
1

Hello,

Is it possible to load a pretrained Bert model but with a different type_vocab_size value?

For exmaple.:

  • The pretrained model: stsb-bert-base

  • The config.json of the stsb-bert-base model:
    image

  • Now, I want to set the type_vocab_size to 20, is there a possible way that I can still use the pretrained stsb-bert-base model as a start point to fine-tune on my own dataset? Or I have to train the model from scratch?

  • As a reference, I tried to loading the pretrained model with a modified type_vocab_size=20, but got an error as below:
    “RuntimeError: Error(s) in loading state_dict for BertModel: size mismatch for bert.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([20, 768]).”

Thank you! :sunflower:
Br,
Na