Hello! I can’t figure out why the positional embeddings are implemented as just the vanilla Embedding layer in both PyTorch and Tensorflow . Based on my current understanding, positional embeddings should be implemented as non-trainable sin/cos or axial positional encodings (from reformer). Can any…

Why positional embeddings are implemented as just simple embeddings?

vgoklani August 4, 2020, 5:00pm 3

Thanks @yjernite - curious, what are the pros and cons of both approaches? When should one choose to use trigonometric functions vs applying an nn.Embedding on an ordinal index?

1 Like

Topic		Replies	Views
`BertEmbeddings` contains positional embedding? 🤗Transformers	2	3162	December 27, 2022
Positional Embeddings in Transformer Implementations 🤗Transformers	1	1810	September 3, 2024
What are the goals in Positional Embedding methods? 🤗Transformers	2	505	March 3, 2022
Creating a tokenizer with both custom tokens and positions Beginners	5	1236	April 22, 2022
Issues with Whisper Encoder: Positional Encoding 🤗Transformers	4	1586	November 16, 2022

Why positional embeddings are implemented as just simple embeddings?

Related topics