Trainer API showing loss = 0 after first log

I am trying to make a custom AI model compatible with the trainer API and it seems that the loss is acting in an expected behaviour, there is no error or anything, just this one :

Code is pretty long to paste here so I uploaded a copy here :

If any ideas why this is happeninig ?