|
Questions on model's tokens
|
|
0
|
134
|
March 24, 2021
|
|
Token Classification: How to tokenize and align labels with overflow and stride?
|
|
1
|
393
|
March 15, 2021
|
|
Space token ' ' cannot be add when is_split_into_words = True
|
|
1
|
77
|
March 11, 2021
|
|
Are special_tokens the only tokens guaranteed to be atomic?
|
|
0
|
94
|
March 3, 2021
|
|
Does AutoTokenizer.from_pretrained add [cls] tokens?
|
|
7
|
240
|
March 2, 2021
|
|
BertTokenizer's encode_plus returns 2d tensor when printing 'input_ids'/ 'attention_mask'
|
|
0
|
65
|
February 7, 2021
|
|
Combine multiple sentences together during tokenization
|
|
2
|
156
|
January 29, 2021
|
|
āOSError: Model name './XX' was not found in tokenizers model name listā - cannot load custom tokenizer in Transformers
|
|
12
|
775
|
January 25, 2021
|
|
Tunning tokenizer on my own dataset
|
|
0
|
134
|
January 25, 2021
|
|
Import distilbert-base-uncased tokenizer to an android app along with the tflite model
|
|
0
|
95
|
January 16, 2021
|
|
ERROR?why encoding [MASK] before '.' would gain a idx 13?
|
|
4
|
119
|
December 28, 2020
|
|
Why Bert-chinese use do_lower_case=False?
|
|
0
|
88
|
December 24, 2020
|
|
Bug with tokernizer's offset mapping for NER problems?
|
|
3
|
412
|
December 23, 2020
|
|
BERT WordPiece Tokenizer: some matras missing after tokenization for Hindi Language #572
|
|
0
|
71
|
December 23, 2020
|
|
Error with new tokenizers (URGENT!)
|
|
5
|
1107
|
December 16, 2020
|
|
Error with <|endoftext|> in Tokenizer GPT2
|
|
2
|
197
|
December 16, 2020
|
|
Build a RoBERTa tokenizer from scratch
|
|
5
|
397
|
December 12, 2020
|
|
Issue with post-processing
|
|
0
|
113
|
December 8, 2020
|
|
Couldn't instantiate the backend tokenizer
|
|
0
|
336
|
December 7, 2020
|
|
OSError: Model name 'gpt2' was not found in tokenizers model name list (gpt2,...)
|
|
5
|
534
|
November 24, 2020
|
|
Bypassing tokenizers
|
|
2
|
87
|
November 23, 2020
|
|
Tokenizing Domain Specific Text
|
|
5
|
222
|
November 20, 2020
|
|
Tokenizer splits up pre-split tokens
|
|
4
|
238
|
November 18, 2020
|
|
Issue with tokenizer.tokenize
|
|
3
|
124
|
November 16, 2020
|
|
Tokenizer taking extremely long time to train
|
|
0
|
99
|
November 3, 2020
|
|
Where to find the "wiki-big.train.raw" data as mentioned in the snippet for tokenizers 0.9?
|
|
2
|
101
|
October 29, 2020
|
|
Change bpe-dropout value on the fly?
|
|
0
|
107
|
October 24, 2020
|
|
Loading pretrained SentencePiece tokenizer from Fairseq
|
|
5
|
925
|
October 21, 2020
|
|
Speed up Longformer Tokenizer
|
|
2
|
151
|
October 18, 2020
|
|
What does `tokenizers.normalizer.normalize` do?
|
|
5
|
371
|
October 12, 2020
|
