What is the recommended way of re-initializing only the decoder weights of a pretrained model?
In the case of Mask2Former I noticed that it has a different weight initializing method depending on the layer.
from transformers import AutoModel
mask2former_model = AutoModel.from_pretrained("facebook/mask2former-swin-tiny-cityscapes-panoptic")
# re-initialize decoder weights...