|
Is it okay to split ids sequence when it is encoded using Byte-level BPE
|
|
0
|
65
|
July 7, 2021
|
|
Using truncated fragments as input samples in training
|
|
3
|
155
|
July 1, 2021
|
|
Using whitespace tokenizer for training models
|
|
1
|
110
|
June 6, 2021
|
|
Sentence splitting
|
|
6
|
2646
|
June 3, 2021
|
|
Save custom components
|
|
0
|
101
|
May 29, 2021
|
|
Regular tokens vs special tokens
|
|
3
|
161
|
May 26, 2021
|
|
How to see contents of a normalizer
|
|
0
|
92
|
May 7, 2021
|
|
Newbie: Main difference between tokenizers?
|
|
0
|
136
|
May 6, 2021
|
|
Can't load tokenizer for 'sshleifer/student_blarge_12_3'
|
|
0
|
95
|
May 6, 2021
|
|
How to create a Huggingface tokenizer from a non-Huggingface tokenizer?
|
|
0
|
107
|
May 4, 2021
|
|
Add new tokens and learn the embeddings of the new tokens and keeping all the other parametes frozen
|
|
0
|
109
|
April 30, 2021
|
|
How to tokenize large contexts without running out of memory
|
|
0
|
269
|
April 30, 2021
|
|
How do you use SentencePiece for BPE of sequences with no whitespace
|
|
1
|
394
|
April 29, 2021
|
|
BOS tokens for mBERT tokenizer
|
|
1
|
128
|
April 14, 2021
|
|
Issues with offset_mapping values
|
|
3
|
344
|
April 10, 2021
|
|
BertTokenizerFast for stsb-xlm-r-multilingual model
|
|
3
|
160
|
April 8, 2021
|
|
Skip-gram tokens
|
|
0
|
123
|
April 4, 2021
|
|
How to add additional custom pre-tokenization processing?
|
|
5
|
964
|
March 30, 2021
|
|
Get vocabulary tokens in order to exclude them from generate function
|
|
0
|
135
|
March 29, 2021
|
|
Using a BertWordPieceTokenizer trained from scratch from transformers
|
|
2
|
298
|
March 26, 2021
|
|
Questions on model's tokens
|
|
0
|
164
|
March 24, 2021
|
|
Token Classification: How to tokenize and align labels with overflow and stride?
|
|
1
|
687
|
March 15, 2021
|
|
Space token ' ' cannot be add when is_split_into_words = True
|
|
1
|
102
|
March 11, 2021
|
|
Are special_tokens the only tokens guaranteed to be atomic?
|
|
0
|
123
|
March 3, 2021
|
|
Does AutoTokenizer.from_pretrained add [cls] tokens?
|
|
7
|
476
|
March 2, 2021
|
|
BertTokenizer's encode_plus returns 2d tensor when printing 'input_ids'/ 'attention_mask'
|
|
0
|
84
|
February 7, 2021
|
|
Combine multiple sentences together during tokenization
|
|
2
|
376
|
January 29, 2021
|
|
āOSError: Model name './XX' was not found in tokenizers model name listā - cannot load custom tokenizer in Transformers
|
|
12
|
1416
|
January 25, 2021
|
|
Tunning tokenizer on my own dataset
|
|
0
|
187
|
January 25, 2021
|
|
Import distilbert-base-uncased tokenizer to an android app along with the tflite model
|
|
0
|
156
|
January 16, 2021
|
