Weight duplication/reuse for small model config changes?

Hello everyone,

A huge thank you in advance for your help as always. I am working on an application involving a ViT model. I would like to change the image size (and patch size) of the model to something that is not available pre-trained. Is there an easy way to load in pre-trained weights and then copy them to the new weights that are required when the image/patch size (i.e. transformer sequence length) is changed? Figure starting from this is better than completely starting from scratch weights.

Thanks again.