|
Introducing FlashTokenizer: The World's Fastest Tokenizer Library for LLM Inference
|
|
2
|
4
|
March 21, 2025
|
|
Why does tokenization take so long?
|
|
1
|
222
|
March 19, 2025
|
|
Call rust function in python
|
|
1
|
14
|
March 19, 2025
|
|
Tokenizer taking extremely long time to train
|
|
1
|
929
|
March 19, 2025
|
|
Rs-bpe tokenizer [PyPI | Python] - Outperforms tiktoken & tokenizers
|
|
2
|
7
|
March 19, 2025
|
|
Build failure with NoGIL Python on main
|
|
0
|
7
|
March 17, 2025
|
|
SentencePiece to Tokenizers conversion
|
|
0
|
11
|
March 14, 2025
|
|
Decode token IDs into a list (not a single string)
|
|
4
|
3386
|
March 11, 2025
|
|
Adapter-aware chat_template
|
|
3
|
103
|
February 21, 2025
|
|
Fine-tuning whsiper on custom special tokens
|
|
0
|
27
|
February 16, 2025
|
|
Inconsistent, mojibaked tokenization in some but not all Huggingface tokenizers
|
|
1
|
13
|
February 12, 2025
|
|
Errors with Tokenizers on Llama
|
|
2
|
158
|
February 8, 2025
|
|
Character level tokenizer with specific order
|
|
5
|
34
|
February 7, 2025
|
|
Custom Tokenizer Error - Please Help!
|
|
0
|
16
|
February 7, 2025
|
|
ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['pixel_values']
|
|
0
|
49
|
February 4, 2025
|
|
BPEtokenizer reports error "not valid UTF-8" when processing txt file
|
|
7
|
32
|
January 29, 2025
|
|
NLLB tokenizer multiple target/source languages within a training batch
|
|
5
|
1305
|
January 10, 2025
|
|
Creating a Custom Token Vocabulary for GPT-2
|
|
1
|
139
|
January 7, 2025
|
|
'NoneType' Object Error and Token Authorization Issue in OpenAI API Integration
|
|
4
|
82
|
January 5, 2025
|
|
Train_from_iterator throwing TypeError: expected string or buffer errir
|
|
2
|
20
|
January 3, 2025
|
|
What should be processing_class param value of Seq2SeqTrainer for VisionEncoderDecoderModel Finetuning?
|
|
0
|
19
|
January 3, 2025
|
|
Error creating custom pre_tokenizer
|
|
3
|
29
|
January 2, 2025
|
|
Adding special tokens to LEDTokenizer
|
|
0
|
35
|
December 25, 2024
|
|
Byte Level Tokenizer While Training
|
|
0
|
34
|
December 14, 2024
|
|
Tokenizer: what function removes spaces between '<' and '>'?
|
|
0
|
43
|
December 9, 2024
|
|
Convert huggingface tokenizer into sentencepiece format
|
|
1
|
511
|
November 27, 2024
|
|
Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error
|
|
6
|
97
|
November 6, 2024
|
|
Generate tokenizer.json for Marian(Opus) MT
|
|
2
|
634
|
November 4, 2024
|
|
Tokenizer method inference
|
|
3
|
31
|
November 2, 2024
|
|
How to skip tokens from translation?
|
|
2
|
871
|
October 15, 2024
|