|
Issue with post-processing
|
|
0
|
143
|
December 8, 2020
|
|
Couldn't instantiate the backend tokenizer
|
|
0
|
408
|
December 7, 2020
|
|
OSError: Model name 'gpt2' was not found in tokenizers model name list (gpt2,...)
|
|
5
|
711
|
November 24, 2020
|
|
Bypassing tokenizers
|
|
2
|
106
|
November 23, 2020
|
|
Tokenizing Domain Specific Text
|
|
5
|
256
|
November 20, 2020
|
|
Tokenizer splits up pre-split tokens
|
|
4
|
406
|
November 18, 2020
|
|
Issue with tokenizer.tokenize
|
|
3
|
154
|
November 16, 2020
|
|
Tokenizer taking extremely long time to train
|
|
0
|
127
|
November 3, 2020
|
|
Where to find the "wiki-big.train.raw" data as mentioned in the snippet for tokenizers 0.9?
|
|
2
|
113
|
October 29, 2020
|
|
Change bpe-dropout value on the fly?
|
|
0
|
120
|
October 24, 2020
|
|
Loading pretrained SentencePiece tokenizer from Fairseq
|
|
5
|
1125
|
October 21, 2020
|
|
Speed up Longformer Tokenizer
|
|
2
|
171
|
October 18, 2020
|
|
What does `tokenizers.normalizer.normalize` do?
|
|
5
|
441
|
October 12, 2020
|
|
Automatic sentence segmentation and encoding
|
|
0
|
372
|
October 12, 2020
|
|
How to truncate from the head in AutoTokenizer?
|
|
2
|
757
|
September 26, 2020
|
|
How much memory is needed for training ByteLevelBPETokenizer?
|
|
3
|
557
|
September 18, 2020
|
|
How to make tokenizer convert subword token to an independent token?
|
|
4
|
232
|
September 9, 2020
|
|
How to know if a subtoken is a word or part of a word?
|
|
8
|
1003
|
September 1, 2020
|
|
Using a pretrained tokenizer vs training a one from scratch
|
|
1
|
236
|
August 21, 2020
|
|
Masking Probability
|
|
4
|
171
|
August 20, 2020
|
|
Tokenizer not found
|
|
0
|
110
|
August 18, 2020
|
|
Add new tokens for subwords
|
|
9
|
2124
|
August 11, 2020
|
|
Token alignment for word-level tasks
|
|
1
|
613
|
August 5, 2020
|
|
ByteLevelBPETokenizer inconsistent behavior
|
|
0
|
149
|
July 23, 2020
|
|
Use a pretrained ByteLevelBPETokenizer on text
|
|
1
|
864
|
July 17, 2020
|
|
Mask only specific words
|
|
2
|
490
|
July 11, 2020
|
|
Continuation token in pertained tokenizer bert-base-chinese
|
|
0
|
211
|
July 11, 2020
|
|
Tokenizers v0.8.0 is out!
|
|
0
|
625
|
July 7, 2020
|
|
About the Tokenizers category
|
|
1
|
117
|
July 7, 2020
|
