Fine Tuning Segformer on Custom Dataset, getting negative loss

I am fine tuning a segformer model from Hugging Face on my custom dataset, I am seeing negative losses. The predictions were completely off too. What am I missing?

I am following this blog: Fine-Tune a Semantic Segmentation Model with a Custom Dataset

Related question, where is the mapping for annotation’s color to the label is supposed to be done.
In my case there is just 1 label whose pixel value in the mask image is 254 (gray scale) and everything else is 100. However, how does the model know what is what?