Loading pretrained weights into model for sequence classifcation

Beginners
1

Hi there,

I am a bit confused as to how saving models works. I have been pretraining with an ElectraForPreTraining model and have saved it using save_pretrained(). However, I am now interested in fine-tuning the model on a sequence classification task, so I figure I would have to load my pretrained weights into an ElectraForSequenceClassification model, except the ElectraForPreTraining model has a different head on the discriminator than the ElectraForSequenceClassification and I am not sure how to handle that correctly. Does the from_pretrained() method handle this out of the box and if not what would I have to do?

2

Hi @rsvarma the .from_pretrained method handles this. You can do

ElectraForSequenceClassification.from_pretrained("path to your pre-trained electra model")
1 Like
3

Hi @valhalla,

Thanks, thats what I needed to know!