How is CLS special token embedding initialized?

BramVanroy · March 16, 2022, 3:46pm

Just like other tokens, the CLS token is randomly initialized from a normal distribution. The only exception is the padding token, which is set to zero.

github.com

huggingface/transformers/blob/204c54d411c2b4c7f31405203533a51632f46ab1/src/transformers/models/bert/modeling_bert.py#L731-L734

      
        
            elif isinstance(module, nn.Embedding):
                module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
                if module.padding_idx is not None:
                    module.weight.data[module.padding_idx].zero_()

Topic		Replies	Views
How to add a new special token and initialize its embeddings to random values? Beginners	0	275	October 19, 2022
DistilBERT and CLS token Beginners	2	2431	February 21, 2021
Should cls_token be [CLS] or <cls>? 🤗Tokenizers	3	273	October 11, 2023
Identical CLS token embeddings for all different sentences? Beginners	1	450	April 17, 2023
Special tokens with inputs_embeds input Beginners	0	260	July 10, 2023

How is CLS special token embedding initialized?

Related topics