Are Word Embeddings by BERT generated for long sequences better than ones generated for short sequences?

frap · March 29, 2022, 5:04pm

I would like to work with token embeddings of words. Does longer sentential contexts (like paragraphs of 500 tokens long) improve word embeddings compared to the ones generated for shorter sequences?

Topic		Replies	Views
Creating word embeddings using BERT of machine generated sequential data Models	0	256	April 7, 2023
Word, sentence or long context embedding? Beginners	0	318	March 8, 2024
Training BERT for word embedding Beginners	17	13510	November 12, 2022
Fine-tuning BERT with sequences longer than 512 tokens Models	7	25050	April 4, 2022
Combining vectors when using contextual word embeddings with large datasets Beginners	0	18	July 23, 2024

Are Word Embeddings by BERT generated for long sequences better than ones generated for short sequences?

Related topics