|
STEM image/text based PDF textbooks for extraction, indexing and embeddings
|
|
2
|
122
|
September 2, 2026
|
|
ValueError loading Helsinki-NLP tokenizers
|
|
3
|
141
|
March 12, 2026
|
|
Doesn't gpt-oss use tiktoken?
|
|
2
|
431
|
February 10, 2026
|
|
XLM-R vs llama-7b tokenization
|
|
5
|
249
|
February 6, 2026
|
|
Training a tokenizer clarification question
|
|
1
|
130
|
December 4, 2025
|
|
Tokenizer dataset is very slow
|
|
4
|
4761
|
November 25, 2025
|
|
Return_offsets_mapping when decoding
|
|
4
|
422
|
November 20, 2025
|
|
Training a Vision Language Model on Text Only Data
|
|
2
|
101
|
October 11, 2025
|
|
Advice for Singing Synthesis and Tokenization
|
|
2
|
73
|
October 8, 2025
|
|
Intelligent Tokenizer: Attention Needs No Vocabulary (Demo + Paper)
|
|
0
|
80
|
September 14, 2025
|
|
Qwen tokenizer, omit Ä from offset_mapping
|
|
3
|
217
|
July 18, 2025
|
|
Word level tokenizer pulls special tokens out of pretokenized strings
|
|
3
|
83
|
July 4, 2025
|
|
Adding atomic / indivisible tokens to BPE tokenizer
|
|
8
|
331
|
July 3, 2025
|
|
Tokenizer is splitting special token
|
|
3
|
112
|
June 30, 2025
|
|
How to determine if a token is special
|
|
2
|
259
|
April 29, 2025
|
|
Saving tokens by system prompt
|
|
1
|
264
|
April 22, 2025
|
|
Introducing FlashTokenizer: The World's Fastest Tokenizer Library for LLM Inference
|
|
0
|
96
|
March 21, 2025
|
|
Why does tokenization take so long?
|
|
1
|
733
|
March 19, 2025
|
|
Call rust function in python
|
|
1
|
93
|
March 19, 2025
|
|
Tokenizer taking extremely long time to train
|
|
1
|
1064
|
March 19, 2025
|
|
Rs-bpe tokenizer [PyPI | Python] - Outperforms tiktoken & tokenizers
|
|
0
|
237
|
March 19, 2025
|
|
Build failure with NoGIL Python on main
|
|
0
|
58
|
March 17, 2025
|
|
SentencePiece to Tokenizers conversion
|
|
0
|
309
|
March 14, 2025
|
|
Decode token IDs into a list (not a single string)
|
|
4
|
5073
|
March 11, 2025
|
|
Adapter-aware chat_template
|
|
3
|
228
|
February 21, 2025
|
|
Fine-tuning whsiper on custom special tokens
|
|
0
|
225
|
February 16, 2025
|
|
Inconsistent, mojibaked tokenization in some but not all Huggingface tokenizers
|
|
1
|
77
|
February 12, 2025
|
|
Errors with Tokenizers on Llama
|
|
2
|
444
|
February 8, 2025
|
|
Character level tokenizer with specific order
|
|
5
|
198
|
February 7, 2025
|
|
Custom Tokenizer Error - Please Help!
|
|
0
|
63
|
February 7, 2025
|