The best way to load pertained weights then continue training BERT?

What is the best way to load pertained weights then continue training BERT using my server’s GPU? Should I load the model as usual then use the Trainer()? The tutorial for continue training seems to be rare.

I’m working on prajjwal1/bert-tiny (now) and bert-base-uncased (next step).

yes, you load the model from .from_pretrained(), and use trainer to train. The link below has examples and video tutorials

1 Like