I am doing Text Classification with Byt5 model. I am pre-training with Simple Transformers library.
Byt5 model always predicts only 1 class as output, giving very low accuracy, no matter how much ever hyper-parameter tuning I do. Same use-case with T5 or MT5 giving good results.
Have anyone else faced the issue with byt5 model?