How to manually change the head of a fine-tuned model that doesn't work with AutoModelFor*?

Hi everyone,

How can I manually remove the head of a checkpoint that was fine-tuned for a specific task, and then add my own head for another task?

Let’s say I want to use the google/bigbird-pegasus-large-arxiv checkpoint that has been fine-tuned for summarization on arxiv dataset. I cannot load it using AutoModelForTokenClassification as Big Bird Pegasus doesn’t have the TokenClassification class.

Can anyone provide a generalizable solution to this?

Thank you very much!