How do I finetune the backbone of bert-base-uncased?

Hey there, I am interested in finetuning bert-base-uncased, I’m talking about finetuning the backbone of the bert, not finetuning it for a specific task. i.e. I have a legal dataset and want to make a legal-bert-base-uncased.

I found this tutorial Fine-tune a pretrained model, but it focuses on finetuning a prediction head rather than the backbone weights.

I would like to

  1. finetune the backbone weights here, by dumping large corpus of texts from my domain,
  2. train a prediction head with a more limited dataset from my domain
    is that possible?