Hi all,
Is anyone aware of plans to implement deBERTa v3 as a distinct model in HuggingFace in the same way that the original deBERTa and deBERTa-v2 have been implemented?
I am aware that there are pretrained v3 models that can be finetuned, but I am interested in training from scratch as this model seems to be a significant improvement on previous iterations (especially the RTD mechanism for training).
5 Likes
Did you find this ?
1 Like
No, I don’t think there’s enough interest in porting this. You could consider ModernBERT as an alternative.
2 Likes
do you have a pretraining code for modernbert ?
1 Like
Check here
1 Like
I read a research paper and they made an observation that DeBERTa is better than ModernBERT in low-resources language contexts. Have you looked into this?
1 Like