Hello, I have a few questions about the BertModelLMHeadModel: Is BertModelLMHeadModel used to conduct the regular language modeling (next token prediction), as it is the case for the GPT2LMHeadModel? For GPT2LMHeadModel, I can just specify labels = input_ids for convenience. I just specify t…

Questions on the `BertModelLMHeadModel`

sgugger October 5, 2020, 1:36pm 8

Yes it’s definitely a mistake. Will fix this morning.

Topic		Replies	Views
Use BertLMHeadModel to finetunning a language model 🤗Transformers	0	328	March 30, 2021
Use of "input_ids,token_type_ids and lm_labels" in BERT Language model 🤗Transformers	1	1060	September 20, 2020
Fine-tune BERT for Masked Language Modeling 🤗Transformers	3	3051	January 25, 2021
ELECTRA for Causal LM 🤗Transformers	0	500	April 8, 2021
Where in the code does masking of tokens happen when pretraining BERT Beginners	5	7335	August 17, 2020