I really don’t get what’s the input of BERT. I read a lot of thing about BERT and most of it is a very confusing. I know there are three embedding layers as well as I know the intuition behind each of them. But, what’s exactly a token embedding, a segment embedding, and a positional embedding? What…

Hi! :slight_smile: BERT’s input is essentially subwords. For example, if I want to feed BERT the sentence “Welcome to HuggingFace Forums!”, what I actually gets fed in is: ['[CLS]', 'welcome', 'to', 'hugging', '##face', 'forums', '!', '[SEP]']. Each of these tokens is mapped to an integer: [101,…

What's the input of BERT?

Beginners

frap February 19, 2022, 10:12pm 3

Hi! Thank you very much for your time. Your answer was of great help

1 Like

Topic		Replies	Views
The inputs into BERT are token IDs. How do we get the corresponding input token VECTORS? Beginners	10	17927	September 15, 2022
Bert embedding layer Beginners	1	2752	January 4, 2022
How to get a model's initial input representation? 🤗Transformers	2	848	June 21, 2022
How to access tokens embedding layer in BERT? 🤗Transformers	0	817	April 13, 2021
What should be used as sentence embedding for BertModel? Beginners	0	1915	May 24, 2021

What's the input of BERT?

Related topics