Add a classification head to a fine-tuned language model

We I have fine-tuned a GPT-2 model with a language model head on medical triage text, and would like to use this model as a classifier. However, as far as I can tell, the Automodel Huggingface library allows me to have either a LM or a classifier etc. head, but I don’t see a way to add a classifier on top of a fine-tuned LM.

Am I mistaken in my understanding of the possibilities? We want to preserve the fine-tuning we have done for the LM task, so don’t want to swap this head for a classifier and lose all of those weights. I suppose we could unlock the top layers of the GPT2 model and try and push the training back into the model, but I suspect that would soon become untenable for our somewhat limited resources. I see this question has been asked here, and my fellow worker has posted a follow-up question in that thread, but we didn’t see any answer we could relate to, so I am asking again in this new topic.

You need to use the AutoModelForSequenceClassification class to add a classification head on top of your pretrained model.

Thanks for your reply @sgugger. I have a fine-tuned pretrained model, that is - a AutoModelForCausalLM.from_pretrained("gpt2-large"), which I have fine-tuned on my specific language (which is triage text and quite unlike standard English).

I thought that if I called an AutoModelForSequenceClassification("gpt2-large") then it would be just like reverting to the standard pretrained model, without any of the fine-tuning I have just done, since my fine-tuning resides in the head of the AutoModelForCausalLM.

My question was if it was possible to sit a classifier head on top of the fine tuned language generator model, so as to inherit the weights that I have obtained through the fine-tuning. If I load the weights of the AutoModelForCausalLM into the AutoModelForSequenceClassification then the fine-tuned weights are discarded, but would it be beneficial to add a linear layer to allow those weights to be preserved for use by a manually added classifier head?