|
BOS tokens for mBERT tokenizer
|
|
1
|
26
|
April 14, 2021
|
|
Sentence splitting
|
|
5
|
70
|
April 13, 2021
|
|
Issues with offset_mapping values
|
|
3
|
66
|
April 10, 2021
|
|
BertTokenizerFast for stsb-xlm-r-multilingual model
|
|
3
|
55
|
April 8, 2021
|
|
Skip-gram tokens
|
|
0
|
25
|
April 4, 2021
|
|
How to add additional custom pre-tokenization processing?
|
|
5
|
369
|
March 30, 2021
|
|
Get vocabulary tokens in order to exclude them from generate function
|
|
0
|
28
|
March 29, 2021
|
|
Using a BertWordPieceTokenizer trained from scratch from transformers
|
|
2
|
75
|
March 26, 2021
|
|
WordLevel error: Missing [UNK] token from the vocabulary
|
|
0
|
49
|
March 26, 2021
|
|
Questions on model's tokens
|
|
0
|
68
|
March 24, 2021
|
|
Token Classification: How to tokenize and align labels with overflow and stride?
|
|
1
|
85
|
March 15, 2021
|
|
Space token ' ' cannot be add when is_split_into_words = True
|
|
1
|
31
|
March 11, 2021
|
|
Are special_tokens the only tokens guaranteed to be atomic?
|
|
0
|
48
|
March 3, 2021
|
|
Does AutoTokenizer.from_pretrained add [cls] tokens?
|
|
7
|
47
|
March 2, 2021
|
|
BertTokenizer's encode_plus returns 2d tensor when printing 'input_ids'/ 'attention_mask'
|
|
0
|
32
|
February 7, 2021
|
|
Training sentencePiece from scratch?
|
|
0
|
77
|
February 2, 2021
|
|
Combine multiple sentences together during tokenization
|
|
2
|
57
|
January 29, 2021
|
|
TypeError when loading tokenizer with from_pretrained method for bart-large-mnli model
|
|
0
|
67
|
January 25, 2021
|
|
“OSError: Model name './XX' was not found in tokenizers model name list” - cannot load custom tokenizer in Transformers
|
|
12
|
224
|
January 25, 2021
|
|
Tunning tokenizer on my own dataset
|
|
0
|
62
|
January 25, 2021
|
|
Import distilbert-base-uncased tokenizer to an android app along with the tflite model
|
|
0
|
45
|
January 16, 2021
|
|
ERROR?why encoding [MASK] before '.' would gain a idx 13?
|
|
4
|
95
|
December 28, 2020
|
|
Why Bert-chinese use do_lower_case=False?
|
|
0
|
64
|
December 24, 2020
|
|
Bug with tokernizer's offset mapping for NER problems?
|
|
3
|
92
|
December 23, 2020
|
|
BERT WordPiece Tokenizer: some matras missing after tokenization for Hindi Language #572
|
|
0
|
50
|
December 23, 2020
|
|
Error with new tokenizers (URGENT!)
|
|
5
|
550
|
December 16, 2020
|
|
Error with <|endoftext|> in Tokenizer GPT2
|
|
2
|
94
|
December 16, 2020
|
|
Build a RoBERTa tokenizer from scratch
|
|
5
|
229
|
December 12, 2020
|
|
Issue with post-processing
|
|
0
|
74
|
December 8, 2020
|
|
Couldn't instantiate the backend tokenizer
|
|
0
|
255
|
December 7, 2020
|
