How is CLS special token embedding initialized?

sgavela · March 16, 2022, 11:07am

Hi! Does anyone know how CLS token is initialized in BERT? I mean, let’s say I would like to train a BERT model from scratch (which of course I’m not doing), how should I initialize CLS embedding? Just at random under some distribution such as uniform? How is this done in BERT?

BramVanroy · March 16, 2022, 3:46pm

Just like other tokens, the CLS token is randomly initialized from a normal distribution. The only exception is the padding token, which is set to zero.

github.com

huggingface/transformers/blob/204c54d411c2b4c7f31405203533a51632f46ab1/src/transformers/models/bert/modeling_bert.py#L731-L734

      
        
            elif isinstance(module, nn.Embedding):
                module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
                if module.padding_idx is not None:
                    module.weight.data[module.padding_idx].zero_()

Topic		Replies	Views
DistilBERT and CLS token Beginners	2	1994	February 21, 2021
BERT embeddings for padding token not 0? Models	4	3212	February 14, 2022
How to get a model's initial input representation? 🤗Transformers	2	485	June 21, 2022
What's the input of BERT? Beginners	4	6249	December 23, 2022
How to get [CLS] embeddings from BertForTokenClassification model Beginners	3	11264	November 27, 2023

How is CLS special token embedding initialized?

Related Topics