I am working on a binary prediction classification task, primarily focusing on fine-tuning a BERT model to learn the association between CVEs and CWEs. I’ve structured my task into three phases: first, using a large dataset of CVE and CWE descriptions from 2000 to 2023 for MLM pretraining, which includes 226,572 records; then using the pretrained model weights for fine-tuning on 13,000 CVE records from 2023; and finally, using the fine-tuned model with task-specific weights for inference. For inference, I randomly select 100 pairs of CVEs for the model to predict, only retaining the model weights and setting the model to model.eval()
. However, I’ve encountered an issue where, after transferring the pretrained model to fine-tuning, the evaluation metrics significantly improve (accuracy, precision, recall, f1), and both training and validation loss decrease substantially. Yet, when I use the fine-tuned model weights for inference, the accuracy turns out to be worse than before.
What is the reason for this discrepancy?
Here are the hyperparameters used for pretraining and fine-tuning:
Pretrain Model:
batch_size = 16
num_epochs = 10
learning_rate = 2e-5
eps = 1e-8
beta1 = 0.9
beta2 = 0.99
weight_decay = 0.01
total_steps = num_epochs * len(train_loader)
warmup_steps = total_steps // 10
early_stopping_patience = 2
mask_prob = 0.15
replace_mask_prob = 0.8
random_replace_prob = 0.10
keep_original_prob = 0.10
Train Model:
learning_rate = 2e-5
batch_size = 16
epoch = 10
I hope someone can help address this issue. If more detailed code or data is needed, I am willing to provide it. Thank you.