|
How to ensure that tokenizers never truncate partial words?
|
|
2
|
38
|
January 24, 2022
|
|
How to ensure the `overflow` with `stride` always starts with a full word?
|
|
0
|
21
|
January 24, 2022
|
|
Adding new tokens to a BERT tokenizer - Getting ValueError
|
|
2
|
464
|
January 16, 2022
|
|
Adding token to t5-base vocab does not respect space
|
|
0
|
59
|
January 13, 2022
|
|
How can I change the token id of a special token?
|
|
0
|
102
|
January 6, 2022
|
|
Run_seq2seq_qa.py: Column 3 named labels expected length 1007 but got length 1000
|
|
0
|
113
|
December 30, 2021
|
|
Import distilbert-base-uncased tokenizer to an android app along with the tflite model
|
|
3
|
334
|
December 29, 2021
|
|
What are the equivalent manner for using texts_to_sequences?
|
|
0
|
133
|
December 29, 2021
|
|
ERROR?why encoding [MASK] before '.' would gain a idx 13?
|
|
5
|
286
|
December 27, 2021
|
|
LongFormer tokenizer has the same token_type_ids for sequence pairs
|
|
0
|
166
|
December 20, 2021
|
|
How to know if a subtoken is a word or part of a word?
|
|
9
|
1731
|
December 17, 2021
|
|
Batch encode plus in Rust Tokenizers
|
|
1
|
259
|
December 13, 2021
|
|
Adding new tokens while preserving tokenization of adjacent tokens
|
|
2
|
253
|
December 7, 2021
|
|
Best solution for train tokenizer and MLM from scratch
|
|
0
|
230
|
December 6, 2021
|
|
Implementing custom tokenizer components (normalizers, processors)
|
|
1
|
287
|
November 30, 2021
|
|
Does T5Tokenizer support the Greek language?
|
|
1
|
314
|
November 24, 2021
|
|
How padding in huggingface tokenizer works?
|
|
4
|
316
|
November 22, 2021
|
|
Why we need to add special tokens to tasks other than classification?
|
|
0
|
317
|
November 17, 2021
|
|
How to configure TokenizerFast for AutoTokenizer
|
|
2
|
398
|
November 11, 2021
|
|
What does this warning mean? -overflowing tokens are not returned for the setting you have chosen
|
|
0
|
570
|
November 11, 2021
|
|
How to employ different vocabs for encoder and decoder respectively?
|
|
0
|
341
|
November 9, 2021
|
|
How to use tokenizer.tokenize in Chinese data properly?
|
|
0
|
337
|
November 9, 2021
|
|
Mask only specific words
|
|
4
|
1064
|
November 7, 2021
|
|
ArrowInvalid: Column 3 named attention_mask expected length 1000 but got length 1076
|
|
2
|
663
|
November 5, 2021
|
|
Load custom pretrained tokenizer
|
|
0
|
393
|
October 28, 2021
|
|
Is that possible to embed the tokenizer into the model to have it running on GCP using TensorFlow Serving?
|
|
1
|
549
|
October 21, 2021
|
|
Added Tokens Not Decoding with Spaces
|
|
2
|
383
|
October 19, 2021
|
|
Using Custom Vocab.txt
|
|
0
|
375
|
October 17, 2021
|
|
Tokenizer.encode not returning encodings
|
|
2
|
371
|
October 9, 2021
|
|
HuggingFace BPE Trainer Error - Training Tokenizer
|
|
0
|
412
|
October 8, 2021
|
