Skip-gram tokens

vitali April 4, 2021, 6:02am 1

Is there any way to produce skip-gram tokens with tokenizers library?

Topic	Replies	Views
Create a simple tokenizer 🤗Tokenizers	335	February 14, 2023
ESM Fast Tokenizer Beginners	65	January 15, 2024
Tokenizers v0.8.0 is out! 🤗Tokenizers	1395	July 7, 2020
Train Retry Tokenizer 🤗Tokenizers	207	April 18, 2023
Tokenizer from own vocab 🤗Tokenizers	348	July 11, 2022