Hi, I want to fine-tune Bert for question answering but instead of using the built-in position embeddings (which are the indices of the tokens), I want to add my own positional embedding (for example the sinusoidal positional embedding mentioned in the paper “Attention is all you need”). I understan…

How to use custom positional embedding while fine tuning Bert

JinKing August 19, 2022, 7:42am 2

Did you solve this problem? I’m trying to solve same thing.

`BertEmbeddings` contains positional embedding?

Topic		Replies	Views
`BertEmbeddings` contains positional embedding? 🤗Transformers	2	3147	December 27, 2022
Creating a tokenizer with both custom tokens and positions Beginners	5	1232	April 22, 2022
Bert embedding layer Beginners	1	2748	January 4, 2022
Add per-word embedding from outer source to Bert embedding layer Beginners	0	302	January 14, 2024
New layer in bert embeddings 🤗Transformers	1	683	April 1, 2022