Hello Huggingface, I try to solve a token classification task where the documents are longer than the model’s max length. I modified the tokenize_and_align_labels function from example token classification notebook . I set the tokenizer option return_overflowing_tokens=True and rewrote the function…

Token Classification: How to tokenize and align labels with overflow and stride?

nielsr March 15, 2021, 11:25am 2

Topic		Replies	Views
NER Label tokenization with overflowing tokens 🤗Tokenizers	4	1452	November 6, 2023
Handle overflowing tokens Beginners	0	127	May 29, 2024
Changing Tokenizer's max_length gets weird result Beginners	2	435	May 17, 2022
Pretrained model with stride doesn't predict long text Beginners	1	355	December 1, 2023
How to ensure the `overflow` with `stride` always starts with a full word? 🤗Tokenizers	0	1277	January 24, 2022