such as I got a string and a tokenizer like: from transformers import AutoTokenizer string = “今天天气真好” tokenizer = AutoTokenizer(“unsloth/Qwen3-14B”) then I want to get something like: [‘今天’, ‘天气’, ‘真’, ‘好’] if I use: input_ids = tokenizer(string)[“input_ids”] print(tokenizer.decode(input_ids)) …

I personally think that just as there is no perfect natural language—let alone programming language—for all purposes, there is no perfect tokenizer for tokenizing it either… Well, there may be safer options or flawed ones. Yes. ChatGPT (and almost every LLM you use) always runs text through a toke…

How can I get a list of word segmentation results for non-English string?

🤗Transformers

system Closed November 7, 2025, 3:37am 15

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Decode token IDs into a list (not a single string) 🤗Tokenizers	4	4757	March 11, 2025
Byte Level Tokenizer While Training 🤗Tokenizers	0	79	December 14, 2024
Inconsistent, mojibaked tokenization in some but not all Huggingface tokenizers 🤗Tokenizers	1	33	February 12, 2025
Show Submodels of PegasusTokenizer 🤗Tokenizers	1	647	April 28, 2022
Use a pretrained ByteLevelBPETokenizer on text 🤗Tokenizers	1	3976	July 17, 2020

How can I get a list of word segmentation results for non-English string?

Related topics