|
OSError: Model name 'gpt2' was not found in tokenizers model name list (gpt2,...)
|
|
5
|
270
|
November 24, 2020
|
|
Bypassing tokenizers
|
|
2
|
73
|
November 23, 2020
|
|
Tokenizing Domain Specific Text
|
|
5
|
186
|
November 20, 2020
|
|
Tokenizer splits up pre-split tokens
|
|
4
|
132
|
November 18, 2020
|
|
Issue with tokenizer.tokenize
|
|
3
|
99
|
November 16, 2020
|
|
How do you use SentencePiece for BPE of sequences with no whitespace
|
|
0
|
108
|
November 4, 2020
|
|
Tokenizer taking extremely long time to train
|
|
0
|
75
|
November 3, 2020
|
|
Where to find the "wiki-big.train.raw" data as mentioned in the snippet for tokenizers 0.9?
|
|
2
|
85
|
October 29, 2020
|
|
Change bpe-dropout value on the fly?
|
|
0
|
76
|
October 24, 2020
|
|
Loading pretrained SentencePiece tokenizer from Fairseq
|
|
5
|
537
|
October 21, 2020
|
|
Speed up Longformer Tokenizer
|
|
2
|
116
|
October 18, 2020
|
|
What does `tokenizers.normalizer.normalize` do?
|
|
5
|
253
|
October 12, 2020
|
|
Automatic sentence segmentation and encoding
|
|
0
|
142
|
October 12, 2020
|
|
How to truncate from the head in AutoTokenizer?
|
|
2
|
349
|
September 26, 2020
|
|
How much memory is needed for training ByteLevelBPETokenizer?
|
|
3
|
318
|
September 18, 2020
|
|
How to make tokenizer convert subword token to an independent token?
|
|
4
|
185
|
September 9, 2020
|
|
How to know if a subtoken is a word or part of a word?
|
|
8
|
473
|
September 1, 2020
|
|
Using a pretrained tokenizer vs training a one from scratch
|
|
1
|
192
|
August 21, 2020
|
|
Masking Probability
|
|
4
|
120
|
August 20, 2020
|
|
Tokenizer not found
|
|
0
|
78
|
August 18, 2020
|
|
Add new tokens for subwords
|
|
9
|
974
|
August 11, 2020
|
|
Token alignment for word-level tasks
|
|
1
|
337
|
August 5, 2020
|
|
ByteLevelBPETokenizer inconsistent behavior
|
|
0
|
119
|
July 23, 2020
|
|
Use a pretrained ByteLevelBPETokenizer on text
|
|
1
|
633
|
July 17, 2020
|
|
Mask only specific words
|
|
2
|
368
|
July 11, 2020
|
|
Continuation token in pertained tokenizer bert-base-chinese
|
|
0
|
174
|
July 11, 2020
|
|
Tokenizers v0.8.0 is out!
|
|
0
|
344
|
July 7, 2020
|
|
About the Tokenizers category
|
|
1
|
91
|
July 7, 2020
|
