Skip-gram tokens

vitali April 4, 2021, 6:02am 1

Is there any way to produce skip-gram tokens with tokenizers library?

Topic		Replies	Views
Train Retry Tokenizer 🤗Tokenizers	0	224	April 18, 2023
SentencePieceUnigramTokenizer 🤗Tokenizers	0	700	September 22, 2022
Translate from one tokenizer to another 🤗Tokenizers	0	169	April 15, 2024
Easy way to compare tokenizers 🤗Tokenizers	0	301	May 1, 2023
Show Submodels of PegasusTokenizer 🤗Tokenizers	1	632	April 28, 2022