My input sentence is very long(more than 512). What should I do when I want to fintune model about classify?Thanks

I know I can intercept the first 512 lengths.But I don’t want to do this. This task can understand the classification of texts.

AutoModelForSequenceClassification can be directly used for classification, but the question is whether the input can be some 512 length sentences, and use pooler layer to classify ?

Or what should I do? Thanks

Hey @ccfeidao you might want to try one of the dedicated models like LongFormer or BigBird which have a longer context size of around 4,096 tokens. See this thread for more details :slight_smile:

1 Like

Thanks ,I will try

1 Like

I want to user BigBird to try my task, but have a new problem, I am very Confusion.