MobileViT Fine tuning

[jpjailson]

Hi,

I’m fine-tuning MobileViT for binary semantic segmentation using a custom dataset with two classes: background and polyp.

I’ve followed several tutorials, including:

• Image Segmentation
• Fine-Tuning a Semantic Segmentation Model on a Custom Dataset and Usage via the Inference API

However, my model achieves high accuracy and IoU only for the background class, while the polyp class consistently gets 0 for both IoU and accuracy.

I’ve already tried setting ignore_index=0 in metric.compute, but the issue persists. Any insights on what might be causing this and how to resolve it?

Thanks in advance!

[paulchenchen]

Hi, I know it has been quite a while since you posted, but I am doing some work on similar applications! Hope you find the solutions already, but have you tried printing out the output of the segmentation? Could the ratio of the background pixels be around 98% and the model is just not classifying the poly pixels at all? In that case, the overall accuracy might still be quite high. I am guessing what you want here is to be able to identify the polyps.

[jpjailson]

Hi, thanks for your reply ! You’re right — the model was only identifying the background. I ran inference on some test images and everything was completely black, which explains the high accuracy for the background class only. However, after switching to a different dataset and training the model for more epochs, it finally started to detect the polyps, although the results are still not very good.