Hi, I want to fine-tune Bert for question answering but instead of using the built-in position embeddings (which are the indices of the tokens), I want to add my own positional embedding (for example the sinusoidal positional embedding mentioned in the paper “Attention is all you need”). I understan…

How to use custom positional embedding while fine tuning Bert

JinKing August 19, 2022, 7:42am 2

Did you solve this problem? I’m trying to solve same thing.

`BertEmbeddings` contains positional embedding?

Topic		Replies	Views
Creating a tokenizer with both custom tokens and positions Beginners	5	939	April 22, 2022
`BertEmbeddings` contains positional embedding? 🤗Transformers	2	2233	December 27, 2022
How to use additional input features for NER? Beginners	27	13588	June 5, 2023
Bert embedding layer Beginners	1	2400	January 4, 2022
Add per-word embedding from outer source to Bert embedding layer Beginners	0	139	January 14, 2024