I am trying to train a ModernBERT model using the HF stack. Since this is for a new language, I will also train the tokenizer (which is the easy part). Since ModernBERT uses dynamic padding and global/local attention, I haven’t seen examples of such support in HF. Am I missing something or should I …

ModernBERT Pretraining using HuggingFace API

Padajno March 17, 2025, 8:26am 3

An updates on this?

1 Like