Using GPT's BPE tokenizer for BERT?

dchang · September 14, 2021, 9:41pm

Hello,

I’m training a custom vocab to train a BERT from scratch, and I was wondering if it would make sense to train a GPT-style BPE tokenizer and use a BertModel.

Has anyone done this kind of training with mismatched tokenizer and model types?
I’d appreciate any insights!

Topic		Replies	Views
I want to use bert model weight to train a gpt model how is that possible 🤗Transformers	0	106	November 4, 2023
BERT and GPT2 embedding questions Beginners	2	1132	December 28, 2022
Questions about the connection between tokenizer and the model Beginners	0	204	September 19, 2023
Using bert tokenizer in Electra model 🤗Transformers	0	321	September 27, 2021
Do you have to use a model card's accompanying tokenizer? Beginners	1	225	November 4, 2022

Using GPT's BPE tokenizer for BERT?

Related Topics