Shortformer: Better Language Modeling using Shorter Inputs

FL33TW00D December 31, 2020, 10:02am 1

Interesting paper focusing on shorter context windows and improving training speed!

2 Likes

Topic	Replies	Views
Model for big context window Beginners	197	June 30, 2024
Are Word Embeddings by BERT generated for long sequences better than ones generated for short sequences? 🤗Transformers	238	March 29, 2022
Slow speed with large context 🤗Transformers	13	July 24, 2024
Context is all you need. Shifting focus for language modeling Research	144	April 1, 2025
Maybe not to generate a word every time 🤗Transformers	112	April 19, 2023