Hello, I am trying to train an encoder-decoder model that uses ModernBert as the encoder and GPT2 as the decoder. I had hoped that this would be straightforward enough using HF provided classes/trainers for Seq2Seq but have run into an error I have not been able to debug. Currently I do the followi…

Can you please try again? update modular_modernbert -- add inputs_embeds param to ModernBertModel huggingface:main ← jxmorris12:patch-1 opened 06:57PM - 20 Dec 24 UTC (UTC) …

Training ModernBert+GPT2

Beginners

John6666 January 6, 2025, 6:16am 2

It looks like the following will be quicker for questions about ModernBERT.

1 Like

Topic		Replies	Views
ModernBERT MaskedLM nan training loss Models	7	381	January 27, 2025
EnocederDecoder training/prediction with two tokenizers Beginners	1	766	October 22, 2024
How to train an EncoderDecoderModel with different pretrained encoder and decoder? 🤗Transformers	2	404	April 2, 2024
Warm-started encoder-decoder models (Bert2Gpt2 and Bert2Bert) Beginners	11	2448	June 9, 2024
GPT2 Training from scratch in German 🤗Transformers	3	2298	October 3, 2020

Training ModernBert+GPT2

Related topics