I am trying to train a ModernBERT model using the HF stack. Since this is for a new language, I will also train the tokenizer (which is the easy part). Since ModernBERT uses dynamic padding and global/local attention, I haven’t seen examples of such support in HF. Am I missing something or should I …

ModernBERT Pretraining using HuggingFace API

John6666 March 17, 2025, 11:34am 4

This page?

Blog article from last year.