|
Change bpe-dropout value on the fly?
|
|
0
|
9
|
October 24, 2020
|
|
Loading pretrained SentencePiece tokenizer from Fairseq
|
|
5
|
79
|
October 21, 2020
|
|
How to add additional custom pre-tokenization processing?
|
|
2
|
35
|
October 21, 2020
|
|
Speed up Longformer Tokenizer
|
|
2
|
31
|
October 18, 2020
|
|
What does `tokenizers.normalizer.normalize` do?
|
|
5
|
35
|
October 12, 2020
|
|
Automatic sentence segmentation and encoding
|
|
0
|
28
|
October 12, 2020
|
|
How to truncate from the head in AutoTokenizer?
|
|
2
|
95
|
September 26, 2020
|
|
How much memory is needed for training ByteLevelBPETokenizer?
|
|
3
|
62
|
September 18, 2020
|
|
How to make tokenizer convert subword token to an independent token?
|
|
4
|
100
|
September 9, 2020
|
|
How to know if a subtoken is a word or part of a word?
|
|
8
|
136
|
September 1, 2020
|
|
Using a pretrained tokenizer vs training a one from scratch
|
|
1
|
101
|
August 21, 2020
|
|
Masking Probability
|
|
4
|
83
|
August 20, 2020
|
|
Tokenizer not found
|
|
0
|
46
|
August 18, 2020
|
|
Add new tokens for subwords
|
|
9
|
240
|
August 11, 2020
|
|
Token alignment for word-level tasks
|
|
1
|
73
|
August 5, 2020
|
|
ByteLevelBPETokenizer inconsistent behavior
|
|
0
|
80
|
July 23, 2020
|
|
Use a pretrained ByteLevelBPETokenizer on text
|
|
1
|
294
|
July 17, 2020
|
|
Mask only specific words
|
|
2
|
180
|
July 11, 2020
|
|
Continuation token in pertained tokenizer bert-base-chinese
|
|
0
|
115
|
July 11, 2020
|
|
Tokenizers v0.8.0 is out!
|
|
0
|
132
|
July 7, 2020
|
|
About the Tokenizers category
|
|
1
|
56
|
July 7, 2020
|
