Using GPT's BPE tokenizer for BERT?

dchang · September 14, 2021, 9:41pm

Hello,

I’m training a custom vocab to train a BERT from scratch, and I was wondering if it would make sense to train a GPT-style BPE tokenizer and use a BertModel.

Has anyone done this kind of training with mismatched tokenizer and model types?
I’d appreciate any insights!

Topic		Replies	Views
I want to use bert model weight to train a gpt model how is that possible 🤗Transformers	0	149	November 4, 2023
BERT and GPT2 embedding questions Beginners	2	1429	December 28, 2022
Questions about the connection between tokenizer and the model Beginners	0	284	September 19, 2023
Tokenizer vs Model 🤗Tokenizers	0	148	June 24, 2024
Using bert tokenizer in Electra model 🤗Transformers	0	350	September 27, 2021

Using GPT's BPE tokenizer for BERT?

Related topics