|
Performance difference between ByteLevelBPE and Wordpiece tokenizers
|
|
0
|
18
|
September 22, 2021
|
|
Should have a `model_type` key in its config.json
|
|
0
|
22
|
September 20, 2021
|
|
Finetuning GPT-J6B for custom dataset
|
|
0
|
36
|
September 13, 2021
|
|
Using a fixed vocab.txt with AutoTokenizer?
|
|
1
|
23
|
September 13, 2021
|
|
Train wordpiece from scratch
|
|
2
|
46
|
September 9, 2021
|
|
Fine-tuned BERT tokenizer taking too long to load
|
|
0
|
41
|
September 6, 2021
|
|
How to save a fast tokenizer using the transformer library and then load it using Tokenizers?
|
|
6
|
75
|
September 1, 2021
|
|
I set up a different batch_size, but the time of data processing has not changed
|
|
0
|
35
|
September 1, 2021
|
|
Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library
|
|
1
|
70
|
August 30, 2021
|
|
Index of wordpieces (subwords) after tokenization by transformers
|
|
0
|
36
|
August 28, 2021
|
|
WordLevel error: Missing [UNK] token from the vocabulary
|
|
2
|
295
|
August 26, 2021
|
|
A problem about FutureWarning？
|
|
0
|
40
|
August 18, 2021
|
|
Extracting embedding values of NLP pertained models from tokenized strings
|
|
3
|
62
|
August 18, 2021
|
|
Adding new tokens to a BERT tokenizer - Getting ValueError
|
|
0
|
50
|
August 15, 2021
|
|
Tokenizer shrinking recipes
|
|
1
|
263
|
August 12, 2021
|
|
Tokenization in a NER context
|
|
5
|
632
|
August 11, 2021
|
|
Unable to convert output to interpretable format
|
|
0
|
57
|
July 31, 2021
|
|
Training sentencePiece from scratch?
|
|
1
|
257
|
July 27, 2021
|
|
BpeTrainer implementation in Python
|
|
0
|
56
|
July 23, 2021
|
|
MBart50Tokenizer vs XLMRobertaTokenizer
|
|
0
|
67
|
July 19, 2021
|
|
Why multilingual BERT tokenizer doesn't remove accent markers?
|
|
0
|
80
|
July 18, 2021
|
|
Best way to mask a multi-token word when using `.*ForMaskedLM` models
|
|
1
|
116
|
July 13, 2021
|
|
TypeError when loading tokenizer with from_pretrained method for bart-large-mnli model
|
|
1
|
243
|
July 8, 2021
|
|
Is it okay to split ids sequence when it is encoded using Byte-level BPE
|
|
0
|
53
|
July 7, 2021
|
|
Using truncated fragments as input samples in training
|
|
3
|
136
|
July 1, 2021
|
|
ArrowInvalid: Column 3 named attention_mask expected length 1000 but got length 1076
|
|
0
|
95
|
June 16, 2021
|
|
Using whitespace tokenizer for training models
|
|
1
|
91
|
June 6, 2021
|
|
Sentence splitting
|
|
6
|
2177
|
June 3, 2021
|
|
Save custom components
|
|
0
|
90
|
May 29, 2021
|
|
Regular tokens vs special tokens
|
|
3
|
140
|
May 26, 2021
|
