Use of "input_ids,token_type_ids and lm_labels" in BERT Language model

[RichardWang]

Hi @vikasRajashekar,
I assume what you said lm_labels is labels, and -1 is -100. (see docs here)

@vikasRajashekar

1.Lets say I pass the above to the model. Does it mean that thee loss is calculated only for last 4 values of the input_ids?

Yes

@vikasRajashekar

2.Does it mean network tries to learn that first 4 elements of input_ids are part of one sentence and other 4 is part of other sentence and it learns to predict the series?

The model tries to learn to predict the last 4 tokens from the context. The context is all input tokens, includes the last four tokens, even the last four tokens are masked or replaced tokens, they contribute correct position information to context. Any way, all input tokens will be used to predict the last four tokens.