Loading only pre-trained backbone for Mask2Former

🤗Transformers
1

I am using huggingface Mask2Former model and would like to only load the pre-trained Swin backbone weights.

Here is what I am currently doing.

# Load pre-trained configuration (random initialization of weights)
model_config = Mask2FormerConfig.from_pretrained(
    "facebook/mask2former-swin-tiny-cityscapes-semantic"
)

# Cannot perform the below operation as backbone_config and backbone cannot be set
# model_config.use_pretrained_backbone = True
# model_config.backbone = "microsoft/swin-tiny-patch4-window7-224"

# So override the backbone config
model_config.backbone_config = None
model_config.use_pretrained_backbone = True
model_config.backbone = "microsoft/swin-tiny-patch4-window7-224"

The above code results in an error w.r.t shape mismatch as self.encoder.channels in Mask2FormerPixelDecoder object expects
…
(hidden_states_norms): ModuleDict(
(stage1): LayerNorm((96,), eps=1e-05, elementwise_affine=True)
(stage2): LayerNorm((192,), eps=1e-05, elementwise_affine=True)
(stage3): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
(stage4): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
…

How do i ensure that only the backbone has pre-trained weights as opposed to the other blocks like pixel decoder and transformer decoder?

PS:

model_config = Mask2FormerConfig.from_pretrained(
    "facebook/mask2former-swin-tiny-cityscapes-semantic"
)
# Loading pre-trained configurations with randomly initilaized weights
model_config.backbone_config = SwinConfig.from_pretrained("microsoft/swin-tiny-patch4-window7-224", out_features=['stage1', 'stage2', 'stage3', 'stage4'])
# The above method works when matching the dimensions but i cannot load the pre-trained weights.