Hi, I want to fine-tune Bert for question answering but instead of using the built-in position embeddings (which are the indices of the tokens), I want to add my own positional embedding (for example the sinusoidal positional embedding mentioned in the paper “Attention is all you need”). I understan…

How to use custom positional embedding while fine tuning Bert

TimShieh September 14, 2022, 12:53pm 3

All you need is change BertEmbeddings class, you can build a new class and inherit BertEmbeddings.
You can follow the post.(I wrote the solution in it.)

Topic		Replies	Views
Creating a tokenizer with both custom tokens and positions Beginners	5	939	April 22, 2022
`BertEmbeddings` contains positional embedding? 🤗Transformers	2	2231	December 27, 2022
How to use additional input features for NER? Beginners	27	13587	June 5, 2023
Bert embedding layer Beginners	1	2400	January 4, 2022
Add per-word embedding from outer source to Bert embedding layer Beginners	0	139	January 14, 2024

How to use custom positional embedding while fine tuning Bert

Related Topics