Are Word Embeddings by BERT generated for long sequences better than ones generated for short sequences?

frap · March 29, 2022, 5:04pm

I would like to work with token embeddings of words. Does longer sentential contexts (like paragraphs of 500 tokens long) improve word embeddings compared to the ones generated for shorter sequences?

Topic		Replies	Views
Creating word embeddings using BERT of machine generated sequential data Models	0	197	April 7, 2023
Training BERT for word embedding Beginners	17	10165	November 12, 2022
Fine-tuning BERT with sequences longer than 512 tokens Models	7	18935	April 4, 2022
Generate raw word embeddings using transformer models like BERT for downstream process Beginners	9	34540	October 4, 2021
Embeddings of added words Intermediate	1	550	September 9, 2022

Are Word Embeddings by BERT generated for long sequences better than ones generated for short sequences?

Related Topics