I’m trying to do continual MLM & SOP pre-training on my own dataset with a pre-trained ALBERt model, but the model albert-base-v2 only have a pre-trained MLM head, but not SOP head. (I tried both AlbertForPreTraining and AlbertForMaskedLM, and used model.from_pretrained("albert-base-v2") to load pre-trained weight.) Could you help me how to load the weight if it exists? If not, would it be important to do SOP training from scratch with the pre-trained ALBERT (with randomly initialized weight) or MLM would be enough (via run_language_modeling.py)?
1 Like