Hello everyone, I have very long genome sequences where I have to do some classification stuff on top. What I want to try is to use a transformer to predict the next token from the 512 chunk sequence and slide this transformer to the whole sequence and use those to work on top of the whole sequence…

Sliding Transformer into a long sequence

rwheel August 11, 2022, 6:47am 2

I’m not very familiar with your problem but I found a thread in the forum that seems to be similar to yours but for question answering task: Handling long text in BERT for Question Answering

1 Like

Topic		Replies	Views
Handling long text in BERT for Question Answering Beginners	7	10160	March 10, 2022
Make available BERT like models work on longer sequences (flash attention) Models	0	528	September 27, 2023
Padded sequences in language model (like BERT) with LSTM on top Beginners	0	316	September 9, 2022
Using time series for SequenceClassification models 🤗Transformers	2	3409	September 7, 2022
Using transformers (BERT, RoBERTa) without embedding layer Research	8	3501	December 16, 2020