|
Training sentencePiece from scratch?
|
|
1
|
163
|
July 27, 2021
|
|
BpeTrainer implementation in Python
|
|
0
|
15
|
July 23, 2021
|
|
Tokenizer shrinking recipes
|
|
0
|
167
|
July 21, 2021
|
|
MBart50Tokenizer vs XLMRobertaTokenizer
|
|
0
|
27
|
July 19, 2021
|
|
Why multilingual BERT tokenizer doesn't remove accent markers?
|
|
0
|
23
|
July 18, 2021
|
|
Best way to mask a multi-token word when using `.*ForMaskedLM` models
|
|
1
|
74
|
July 13, 2021
|
|
TypeError when loading tokenizer with from_pretrained method for bart-large-mnli model
|
|
1
|
169
|
July 8, 2021
|
|
Is it okay to split ids sequence when it is encoded using Byte-level BPE
|
|
0
|
22
|
July 7, 2021
|
|
Using truncated fragments as input samples in training
|
|
3
|
51
|
July 1, 2021
|
|
WordLevel error: Missing [UNK] token from the vocabulary
|
|
1
|
193
|
June 17, 2021
|
|
ArrowInvalid: Column 3 named attention_mask expected length 1000 but got length 1076
|
|
0
|
34
|
June 16, 2021
|
|
Using whitespace tokenizer for training models
|
|
1
|
43
|
June 6, 2021
|
|
Sentence splitting
|
|
6
|
1030
|
June 3, 2021
|
|
Save custom components
|
|
0
|
50
|
May 29, 2021
|
|
Regular tokens vs special tokens
|
|
3
|
91
|
May 26, 2021
|
|
Tokenization in a NER context
|
|
4
|
337
|
May 7, 2021
|
|
How to see contents of a normalizer
|
|
0
|
55
|
May 7, 2021
|
|
Newbie: Main difference between tokenizers?
|
|
0
|
69
|
May 6, 2021
|
|
Can't load tokenizer for 'sshleifer/student_blarge_12_3'
|
|
0
|
58
|
May 6, 2021
|
|
How to create a Huggingface tokenizer from a non-Huggingface tokenizer?
|
|
0
|
68
|
May 4, 2021
|
|
Add new tokens and learn the embeddings of the new tokens and keeping all the other parametes frozen
|
|
0
|
67
|
April 30, 2021
|
|
How to tokenize large contexts without running out of memory
|
|
0
|
119
|
April 30, 2021
|
|
How do you use SentencePiece for BPE of sequences with no whitespace
|
|
1
|
242
|
April 29, 2021
|
|
BOS tokens for mBERT tokenizer
|
|
1
|
86
|
April 14, 2021
|
|
Issues with offset_mapping values
|
|
3
|
182
|
April 10, 2021
|
|
BertTokenizerFast for stsb-xlm-r-multilingual model
|
|
3
|
108
|
April 8, 2021
|
|
Skip-gram tokens
|
|
0
|
80
|
April 4, 2021
|
|
How to add additional custom pre-tokenization processing?
|
|
5
|
663
|
March 30, 2021
|
|
Get vocabulary tokens in order to exclude them from generate function
|
|
0
|
96
|
March 29, 2021
|
|
Using a BertWordPieceTokenizer trained from scratch from transformers
|
|
2
|
188
|
March 26, 2021
|
