Keep NSP head after BertForPretraining

tzuhsial · February 1, 2022, 6:14pm

I am pretraining with MLM + NSP (BertForPretraining), and finetuning with NSP (BertForNextSentencePrediction).

Is there an elegant way which I can keep the NSP head from the pretrained model?

Thanks!

tzuhsial · February 1, 2022, 6:30pm

Validated that using BertForNextSentencePrediction does indeed keep the weights of NSP head of BertForPreTraining

Topic		Replies	Views
Continual pre-training from an initial checkpoint with MLM and NSP Models	4	4196	September 8, 2021
Next sentence prediction on custom model 🤗Transformers	3	3340	May 14, 2024
Continual pre-training vs. Fine-tuning a language model with MLM 🤗Transformers	5	8360	November 30, 2021
BERT Next Sentence Prediction: How to do predictions? Beginners	5	7174	September 29, 2022
How to train BERT from scratch on a new domain for both MLM and NSP? Models	2	2238	February 6, 2021