Hey there, I am interested in finetuning bert-base-uncased
, I’m talking about finetuning the backbone of the bert, not finetuning it for a specific task. i.e. I have a legal dataset and want to make a legal-bert-base-uncased.
I found this tutorial Fine-tune a pretrained model, but it focuses on finetuning a prediction head rather than the backbone weights.
I would like to
- finetune the backbone weights here, by dumping large corpus of texts from my domain,
- train a prediction head with a more limited dataset from my domain
is that possible?