One question is about the pretrain method in Transformer packge ？

🤗Transformers
1

I am a beginner and need your help very much, thank you

In pretrain, the classical BERT input dimension and output dimension are 768 dimensions, and then the cross entropy is calculated as loss, but why ‘BertForMaskedLM’ output dimension is vacob’s (tokens’) number？

BertForMaskedLM’s outpus:
(decoder): Linear(in_features=768, out_features=20000, bias=True)

BERTs outpus:
(dense): Linear(in_features=768, out_features=768, bias=True)
(activation): Tanh()

1 Like
2 
        logits = language_model_output.logits  # shape: [batch_size, sequence_length, vocab_size]
        predicted_token_ids = torch.argmax(logits, dim=-1)
1 Like