Help with BERT Adapter + LoRA for Multi-Label Classification (301 classes)

[izow]

@John6666

Yeah. I think so. Bugs aside, using LoRA (PEFT) can change the content and quality of learning for better or worse. Especially when pre-training a model from scratch, it is usually safer to do so without LoRA first.

By the way, I thought that bias due to class overlapping might occur. This is unlikely to be a problem when there are few classes, so it may be one of the causes in this case.

Alright, I’ll launch a new training run using only BERT to see how it goes. Thanks for the advice!