|
Implementing custom tokenizer components (normalizers, processors)
|
|
1
|
46
|
November 30, 2021
|
|
Does T5Tokenizer support the Greek language?
|
|
1
|
78
|
November 24, 2021
|
|
How padding in huggingface tokenizer works?
|
|
4
|
82
|
November 22, 2021
|
|
Why we need to add special tokens to tasks other than classification?
|
|
0
|
99
|
November 17, 2021
|
|
How to configure TokenizerFast for AutoTokenizer
|
|
2
|
153
|
November 11, 2021
|
|
What does this warning mean? -overflowing tokens are not returned for the setting you have chosen
|
|
0
|
156
|
November 11, 2021
|
|
How to employ different vocabs for encoder and decoder respectively?
|
|
0
|
112
|
November 9, 2021
|
|
How to use tokenizer.tokenize in Chinese data properly?
|
|
0
|
115
|
November 9, 2021
|
|
Mask only specific words
|
|
4
|
730
|
November 7, 2021
|
|
ArrowInvalid: Column 3 named attention_mask expected length 1000 but got length 1076
|
|
2
|
318
|
November 5, 2021
|
|
Load custom pretrained tokenizer
|
|
0
|
153
|
October 28, 2021
|
|
Is that possible to embed the tokenizer into the model to have it running on GCP using TensorFlow Serving?
|
|
1
|
242
|
October 21, 2021
|
|
Added Tokens Not Decoding with Spaces
|
|
2
|
158
|
October 19, 2021
|
|
Using Custom Vocab.txt
|
|
0
|
143
|
October 17, 2021
|
|
Tokenizer.encode not returning encodings
|
|
2
|
151
|
October 9, 2021
|
|
HuggingFace BPE Trainer Error - Training Tokenizer
|
|
0
|
154
|
October 8, 2021
|
|
There is no 0.11.0 tokenizers in pip
|
|
4
|
186
|
September 30, 2021
|
|
Performance difference between ByteLevelBPE and Wordpiece tokenizers
|
|
0
|
174
|
September 22, 2021
|
|
Should have a `model_type` key in its config.json
|
|
0
|
185
|
September 20, 2021
|
|
Finetuning GPT-J6B for custom dataset
|
|
0
|
181
|
September 13, 2021
|
|
Using a fixed vocab.txt with AutoTokenizer?
|
|
1
|
161
|
September 13, 2021
|
|
Train wordpiece from scratch
|
|
2
|
188
|
September 9, 2021
|
|
Fine-tuned BERT tokenizer taking too long to load
|
|
0
|
257
|
September 6, 2021
|
|
How to save a fast tokenizer using the transformer library and then load it using Tokenizers?
|
|
6
|
248
|
September 1, 2021
|
|
I set up a different batch_size, but the time of data processing has not changed
|
|
0
|
171
|
September 1, 2021
|
|
Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library
|
|
1
|
228
|
August 30, 2021
|
|
Index of wordpieces (subwords) after tokenization by transformers
|
|
0
|
152
|
August 28, 2021
|
|
WordLevel error: Missing [UNK] token from the vocabulary
|
|
2
|
514
|
August 26, 2021
|
|
A problem about FutureWarning？
|
|
0
|
159
|
August 18, 2021
|
|
Extracting embedding values of NLP pertained models from tokenized strings
|
|
3
|
228
|
August 18, 2021
|
