Finetuning model with smaller sequence size and Dmodel

Hi all !

I wanted to know if there is any way to finetune a model but with smaller parameters (for exemple : sequence length from 512 to 64 and dModel from 768 to 256).

I could use the pytorch TransformerEncoder and train my own from scratch but I wanted to know if there were some fine-tunning possible (i.e not starting from scratch).

Thanks in addvance for any help,

Have a great day !