deBERTa v3 implementation in HuggingFace (with RTD training)

Hi all,
Is anyone aware of plans to implement deBERTa v3 as a distinct model in HuggingFace in the same way that the original deBERTa and deBERTa-v2 have been implemented?
I am aware that there are pretrained v3 models that can be finetuned, but I am interested in training from scratch as this model seems to be a significant improvement on previous iterations (especially the RTD mechanism for training).

5 Likes

Did you find this ?

1 Like

No, I don’t think there’s enough interest in porting this. You could consider ModernBERT as an alternative.

2 Likes

do you have a pretraining code for modernbert ?

1 Like

Check here

1 Like

I read a research paper and they made an observation that DeBERTa is better than ModernBERT in low-resources language contexts. Have you looked into this?

1 Like