Training ModernBert+GPT2

Bachstelze · January 10, 2025, 6:33pm

Can you please try again?

github.com/huggingface/transformers

update modular_modernbert -- add inputs_embeds param to ModernBertModel

huggingface:main ← jxmorris12:patch-1

opened 06:57PM - 20 Dec 24 UTC

jxmorris12

+141 -57

# What does this PR do? Hi! Congrats on the release of ModernBERT; it looks ama…zing. I'm interested in using ModernBERT eventually to train a new [Contextual Document Embeddings](https://arxiv.org/abs/2410.02525) model. One desired feature is to pass the contextual and word embeddings together in the second stage, which requires setting the `inputs_embeds` kwarg so that we can pass hidden states directly. This is a feature of typical BERT and other transformer implementations but isn't yet allowed by ModernBERT, so I added it. It's only a few additional lines of code. cc: @warner-benjamin @tomaarsen @orionw @staghado @bclavie @NohTow @ArthurZucker

Topic		Replies	Views
I get a "You have to specify either input_ids or inputs_embeds" error, but I do specify the input ids Beginners	6	21309	October 31, 2021
Tokenizer decoding using BERT, RoBERTa, XLNet, GPT2 Beginners	7	8581	September 21, 2020
GPT-GPT encoder decoder 🤗Transformers	0	290	May 4, 2021
Warm-started encoder-decoder models (Bert2Gpt2 and Bert2Bert) Beginners	11	2512	June 9, 2024
EncoderDecoderModel with Longformer and Bert 🤗Transformers	1	632	February 11, 2021

Training ModernBert+GPT2

Related topics