Hi @lewtun, sorry for the late reply. Thank you for the suggestion! I have been trying that method out with subclassing the PreTrainedModel and using a separate AutoModel as an encoder for the model. I have noticed that my models from_pretrained does not seem to load the pretrained encoder. Is there any documentation/examples for this use case?