|
Issue in loading the saved tokenizer
|
|
1
|
227
|
January 4, 2024
|
|
SentencePiece user_defined_symbols and fast tokenizers
|
|
1
|
1286
|
January 3, 2024
|
|
Many ambiguous unicode characters for trained tokenizer
|
|
0
|
270
|
December 31, 2023
|
|
Skew between mistral prompt in docs vs. chat template
|
|
2
|
1064
|
December 27, 2023
|
|
Tokenizer shrinking recipes
|
|
7
|
1875
|
December 24, 2023
|
|
How to decode with custom pad tokens
|
|
3
|
3910
|
December 22, 2023
|
|
Training sentencePiece from scratch?
|
|
8
|
15810
|
December 19, 2023
|
|
Questions re: Tokenizer pipeline composability / reuse outside of the HF ecosystem
|
|
0
|
198
|
December 18, 2023
|
|
Get intermediate tokens and merges used in tokenization
|
|
0
|
382
|
December 1, 2023
|
|
BertTokenizer.decode not understanding new vocabulary
|
|
0
|
316
|
December 1, 2023
|
|
Tokenizer tend to choose added tokens first rather than token in vocab
|
|
1
|
480
|
November 30, 2023
|
|
Special token printed out as output
|
|
6
|
782
|
November 24, 2023
|
|
[NER][Japanese] labeled segment shorter than token
|
|
0
|
210
|
November 23, 2023
|
|
T5Tokenizer add a whitespace token after added special tokens
|
|
0
|
300
|
November 22, 2023
|
|
Unable to register my own tokenizer
|
|
0
|
174
|
November 21, 2023
|
|
Get Problem with Doubled tokens in NLLB Tokenizer After load new vocab!
|
|
0
|
249
|
November 21, 2023
|
|
Use Unicode blocks in regex (in Replace normalizer)
|
|
1
|
1033
|
November 9, 2023
|
|
How to handle translations one source language to many target sentences for the same language
|
|
0
|
171
|
November 9, 2023
|
|
NER Label tokenization with overflowing tokens
|
|
4
|
1108
|
November 6, 2023
|
|
How to use a trained tokenizer for semantic search?
|
|
0
|
306
|
November 5, 2023
|
|
Unk_token not set after training a BPETokenizer tokenizer
|
|
1
|
523
|
November 1, 2023
|
|
AutoTokenizer is very slow when loading llama tokenizer
|
|
2
|
1636
|
October 31, 2023
|
|
Offset mappings differ for tokenizers
|
|
0
|
908
|
October 30, 2023
|
|
The process for tokenizing concatenated dataset is slow st the end of tokenizing
|
|
0
|
154
|
October 30, 2023
|
|
Batch tokenize (split into tokens, without processing)
|
|
4
|
422
|
October 28, 2023
|
|
Does AutoTokenizer uploads data to HuggingFace
|
|
0
|
192
|
October 25, 2023
|
|
RobertaTokenizer decode and tokenize do not have the same output
|
|
0
|
230
|
October 24, 2023
|
|
Training tokenizers with padding in between tokens
|
|
0
|
334
|
October 19, 2023
|
|
Leaving unknown words untokenized like in OpenMNT
|
|
0
|
227
|
October 18, 2023
|
|
Keeping special chars in translations
|
|
0
|
282
|
October 12, 2023
|