Is there a way to correctly load a pre-trained transformers model without the configuration file?

Beginners
6

I think you should be able to do

model.resize_token_embeddings(30528)

before you load the state dict. The state should then load successfully. However, as you point out, it is likely that they added tokens to the tokenizer, so you should get their tokenizer files as well. Then it would be as simple as:

MODEL_PATH = "./checkpoint.pt"
state_dict = torch.load(MODEL_PATH)["model"]
config = AutoConfig.from_pretrained("./bert_config.json")
tokenizer = <load tokenizer here>
model = BertModel(config)

model.resize_token_embeddings(len(tokenizer)) 

model = BertModel._load_state_dict_into_model(
    model,
    state_dict,
    MODEL_PATH
)[0]

# make sure token embedding weights are still tied if needed
model.tie_weights()

# Set model in evaluation mode to deactivate DropOut modules by default
model.eval()
2 Likes