Load fine-tuned LM without the head?

Beginners
1

Is there a way to load a fine-tuned model without the head? I just need the fine-tuned LM, without the head to use it in another context – I mean the task head.

I used the Trainer API, with a predefined AutoModelFor*.

2

Hi :slight_smile:

Every model that inherits PreTrainedModel has a base_model property. For example, if you have a BertForSequenceClassification model named model, then model.base_model will give you a BertModel, which is exactly BertForSequenceClassification minus the classification layer at the end.

3 Likes
3

Thank you @beneyal

1 Like