Hello, I am trying to train an encoder-decoder model that uses ModernBert as the encoder and GPT2 as the decoder. I had hoped that this would be straightforward enough using HF provided classes/trainers for Seq2Seq but have run into an error I have not been able to debug. Currently I do the followi…

Can you please try again? update modular_modernbert -- add inputs_embeds param to ModernBertModel huggingface:main ← jxmorris12:patch-1 opened 06:57PM - 20 Dec 24 UTC (UTC) …

Training ModernBert+GPT2

Beginners

John6666 January 6, 2025, 6:16am 2

It looks like the following will be quicker for questions about ModernBERT.

1 Like

Topic		Replies	Views
Warm-started encoder-decoder models (Bert2Gpt2 and Bert2Bert) Beginners	11	2504	June 9, 2024
Tutorial: Implementing Transformer from Scratch - A Step-by-Step Guide Show and Tell	5	6857	May 1, 2025
EnocederDecoder training/prediction with two tokenizers Beginners	1	786	October 22, 2024
I get a "You have to specify either input_ids or inputs_embeds" error, but I do specify the input ids Beginners	6	21244	October 31, 2021
ModernBERT Pretraining using HuggingFace API Models	3	282	March 17, 2025

Training ModernBert+GPT2

Related topics