|
Are special_tokens the only tokens guaranteed to be atomic?
|
|
0
|
9
|
March 3, 2021
|
|
Does AutoTokenizer.from_pretrained add [cls] tokens?
|
|
7
|
20
|
March 2, 2021
|
|
How to add additional custom pre-tokenization processing?
|
|
4
|
217
|
February 20, 2021
|
|
BertTokenizer's encode_plus returns 2d tensor when printing 'input_ids'/ 'attention_mask'
|
|
0
|
15
|
February 7, 2021
|
|
Training sentencePiece from scratch?
|
|
0
|
43
|
February 2, 2021
|
|
Combine multiple sentences together during tokenization
|
|
2
|
33
|
January 29, 2021
|
|
TypeError when loading tokenizer with from_pretrained method for bart-large-mnli model
|
|
0
|
38
|
January 25, 2021
|
|
“OSError: Model name './XX' was not found in tokenizers model name list” - cannot load custom tokenizer in Transformers
|
|
12
|
111
|
January 25, 2021
|
|
Tunning tokenizer on my own dataset
|
|
0
|
36
|
January 25, 2021
|
|
Import distilbert-base-uncased tokenizer to an android app along with the tflite model
|
|
0
|
23
|
January 16, 2021
|
|
ERROR?why encoding [MASK] before '.' would gain a idx 13?
|
|
4
|
68
|
December 28, 2020
|
|
Why Bert-chinese use do_lower_case=False?
|
|
0
|
45
|
December 24, 2020
|
|
Bug with tokernizer's offset mapping for NER problems?
|
|
3
|
53
|
December 23, 2020
|
|
BERT WordPiece Tokenizer: some matras missing after tokenization for Hindi Language #572
|
|
0
|
32
|
December 23, 2020
|
|
Error with new tokenizers (URGENT!)
|
|
5
|
330
|
December 16, 2020
|
|
Error with <|endoftext|> in Tokenizer GPT2
|
|
2
|
63
|
December 16, 2020
|
|
Build a RoBERTa tokenizer from scratch
|
|
5
|
157
|
December 12, 2020
|
|
Issue with post-processing
|
|
0
|
55
|
December 8, 2020
|
|
Couldn't instantiate the backend tokenizer
|
|
0
|
217
|
December 7, 2020
|
|
OSError: Model name 'gpt2' was not found in tokenizers model name list (gpt2,...)
|
|
5
|
191
|
November 24, 2020
|
|
Bypassing tokenizers
|
|
2
|
60
|
November 23, 2020
|
|
Tokenizing Domain Specific Text
|
|
5
|
150
|
November 20, 2020
|
|
Tokenizer splits up pre-split tokens
|
|
4
|
105
|
November 18, 2020
|
|
Issue with tokenizer.tokenize
|
|
3
|
88
|
November 16, 2020
|
|
How do you use SentencePiece for BPE of sequences with no whitespace
|
|
0
|
83
|
November 4, 2020
|
|
Tokenizer taking extremely long time to train
|
|
0
|
62
|
November 3, 2020
|
|
Where to find the "wiki-big.train.raw" data as mentioned in the snippet for tokenizers 0.9?
|
|
2
|
68
|
October 29, 2020
|
|
Change bpe-dropout value on the fly?
|
|
0
|
61
|
October 24, 2020
|
|
Loading pretrained SentencePiece tokenizer from Fairseq
|
|
5
|
363
|
October 21, 2020
|
|
Speed up Longformer Tokenizer
|
|
2
|
101
|
October 18, 2020
|
