Penalizing model during training

I have finetuned T5 for a specific task, however, I noticed that the model tends to reuse a certain part of the input for some outputs (this is undesirable). Is there a way I can penalize (teach the model to adjust its behavior) the model during training whenever it does this (I already know how to detect the incorrect behavior, I just need help penalizing the model during finetuning)?

Is it even a good idea to penalize the model under these circumstances, or it is it better to let the model learn from the dataset by itself and naturally avoid this (the dataset shows only examples of the desirable output, no examples with the undesirable)?