Hi, I’m currently training a segmentation model, and this is how I initiate my model and change my backbone before the training:
if config[“model_architecture”].lower() == “upernet”:
model_args[“loss_ignore_index”] = model_args.pop(“semantic_loss_ignore_index”)
del model_args[“image_size”]
MODEL_INITIALIZER = UperNetForSemanticSegmentation
Load the new backbone from the checkpoint
if config[“backbone_checkpoint_model”] is not None:
new_backbone = AutoBackbone.from_pretrained(new_backbone_dir, out_indices=(1,2,3,4))
# Replace the existing backbone with the new one
model.backbone = new_backbone
# Optionally, if you need to adjust the configuration
model.config.backbone_config = new_backbone.config
For the other backbones (convnext, convnextv2) this approach seems to work without any issues, but for swinv2, even though the training runs without any errors (I guess this means at least the dimensions match correctly) swinv2 model does not seem to learn under this setting.
Is this the way how the backbones are usually swapped? or am I doing it wrong?
Any sort of input would be appreciated! Thanks!