Skip-gram tokens

vitali April 4, 2021, 6:02am 1

Is there any way to produce skip-gram tokens with tokenizers library?

Topic		Replies	Views
Train Retry Tokenizer 🤗Tokenizers	0	225	April 18, 2023
About the Tokenizers category 🤗Tokenizers	1	314	July 7, 2020
Use a pretrained ByteLevelBPETokenizer on text 🤗Tokenizers	1	3921	July 17, 2020
Can't load tokenizer for 'sshleifer/student_blarge_12_3' 🤗Tokenizers	0	336	May 6, 2021
Automatic sentence segmentation and encoding 🤗Tokenizers	0	847	October 12, 2020