Try to fine tune SegFormer for binary semantic segmentation but metrics are nan

Models
1

Hi, I’m trying to fine tune SegFormer for binary semantic segmentation. I looked over some quick tutorials like semantic segmentation and fine-tune-segformer but I could not run my case.

I have two classes that are car and background.

id2label = { 0: 'background', 1: 'car' }
label2id = { 'background':0, 'car':1}
num_labels = len(id2label)

First, I got the size mismatch error when I try giving input id2label and label2id to the model variable. Then I convert the ignore_mismatched_sizes=True (I do not know that it is a proper way)

model = SegformerForSemanticSegmentation.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512", id2label=id2label, label2id=label2id, ignore_mismatched_sizes=True)

Then when I try to train the model with mean_iou metric from evaluate library, I got nan values for all metrics with these warnings:

/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:258: RuntimeWarning: invalid value encountered in double_scalars
  all_acc = total_area_intersect.sum() / total_area_label.sum()
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:259: RuntimeWarning: invalid value encountered in divide
  iou = total_area_intersect / total_area_union
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:260: RuntimeWarning: invalid value encountered in divide
  acc = total_area_intersect / total_area_label
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:262: RuntimeWarning: Mean of empty slice
  metrics["mean_iou"] = np.nanmean(iou)
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:263: RuntimeWarning: Mean of empty slice
  metrics["mean_accuracy"] = np.nanmean(acc)
Trainer is attempting to log a value of "[nan]" of type <class 'list'> for key "eval/per_category_iou" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute.
Trainer is attempting to log a value of "[nan]" of type <class 'list'> for key "eval/per_category_accuracy" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute.

Ekran görüntüsü 2023-08-23 122857
I use the same compute_metrics function as second link.

So, I could not train the model for my scenario and I got some points that I could not figured out.

  • I’m not sure what my num_labels parameter should be, 1 or 2.
  • SegFormer image processor turn the masks (labels) into 0 or 255 values. Why is it not 0-1. Wouldn’t it be easier for classification?
  • There is a parameter called ignore_index in metrics.compute part. I could not understand very well, is it possible to be related the errors?
  • In the first link, they are using nvidia/mit-b0 for segformer fine tuning but when I check the models on hugginface/models, it is used for image classification and there is seperate nvidia/segformer-b0-finetuned-ade-512-512. What is the reason that they did not used directly segformer-b0?
  • If segformer is not suitable for binary segmentation, what else can I used for this case (better to be robust ones)?
  • I did not find any configuration for loss function also. Is there anything about this I have to do?

So, I’m struggling with this case for 2 days and I need some help :upside_down_face:. What should I do to use segformer in binary segmentation cases? Thanks in advance :pray:

2

Hi,

I had this issue a couple of days ago. If you care about the performance of the background class labels be sure to set the reduce_labels argument to False and furthermore be sure to set ignore_index to 255 in the metric.compute() method, so that it looks like this

metrics = metric.compute(
            predictions=pred_labels,
            references=labels,
            num_labels=len(id2label),
            ignore_index=255,
            reduce_labels=image_processor.do_reduce_labels,
        )

In the example script ignore_index is set to 0 and this gives you the NaN values in your metrics since you ignore the background class this way.

1 Like