Hi all, Thanks for making this forum! I have a list of tests, one of which apparently happens to be 516 tokens long. I have been using the feature-extraction pipeline to process the texts, just using the simple function: nlp = pipeline('feature-extraction') When it gets up to the long text, I g…

Truncating sequence -- within a pipeline

AlanFeder April 13, 2021, 10:15pm 4

I have not – I just moved out of the “pipeline” framework, and used the building blocks. It wasn’t too bad

1 Like

Topic		Replies	Views
How to specify sequence length when using "feature-extraction" 🤗Transformers	3	1301	April 28, 2021
Tokenizer behaviour with pipeline 🤗Tokenizers	0	930	August 1, 2023
How do I setup a TextClassificationPipeline that truncates token sequences Beginners	0	329	September 29, 2021
Predictions with pipeline fails to truncate test set 🤗Transformers	0	181	January 23, 2024
Out of index error in pipeline Beginners	9	6526	June 22, 2022