I’m trying to run the question answering pipeline on chunks of of a long document. The DistilBertTokenizerFast has good features to support this. It can break a document down into overlapping chunks and it can give you the offsets mapping so that you can match up labels with tokens. But in the pipel…

How to convert Tokenizer to TokenizerFast?

valhalla September 30, 2020, 2:57pm 2

You can pass tokenizer to the pipeline

pipeline("", tokenizer=your_tok)

Topic		Replies	Views
Difference betweeen DistilBertTokenizerFast and DistilBertTokenizer? 🤗Transformers	2	3267	July 10, 2021
Convert a Python Tokenizer into a TokenizerFast Beginners	0	342	May 20, 2022
Custom DistilBertTokenizer training 🤗Transformers	3	659	November 13, 2020
Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library 🤗Tokenizers	1	1096	August 30, 2021
Use HF tokenizer as a keras layer 🤗Transformers	0	231	March 14, 2023