The accuracy from pretraining is worse than without pretraining

My current task is to classify the association between CVEs and CWEs. However, I’ve noticed that using BertModel.from_pretrained('bert-base-uncased') in the fine-tuning stage results in lower accuracy compared to when I pretrain with more CVE-related descriptions first, and then fine-tune using the pretrained I don’t understand why this is happening as I have ruled out compatibility issues with the model. It’s worth mentioning that in the pretraining phase, I only use the pretrained model weights for fine-tuning, and the tokenizer is consistently BertTokenizer.from_pretrained('bert-base-uncased'). I did not retrain or expand the tokenizer during pretraining because it is very time-consuming.

Here are the hyperparameters I am using:

batch_size = 16
num_epochs = 10
learning_rate = 1e-4
beta1 = 0.9
beta2 = 0.99
weight_decay = 0.01
total_steps = num_epochs * len(train_loader)
warmup_steps = total_steps // 10
early_stopping_patience = 2

Additionally, the settings for masked language modeling (MLM) are:


I hope someone can answer my question. If more detailed code is needed, I can provide it. Thank you.