Tokenizer is splitting special token

John6666 · June 30, 2025, 5:29am

Hmm… How about save_pretrained with from_pretrained way?

github.com/huggingface/tokenizers

PretrainedTokenizerFast from Tokenizer Does Not Keep The Same Properties?

opened 10:19PM - 17 Aug 21 UTC

closed 01:49AM - 31 Mar 24 UTC

FeryET

Stale

I can't seem to create a "PreTrainedTokenizerFast" object from my original `toke…nizers` tokenizer object that has the same proporties. This is the code for a byte pair tokenizer I have experimented on. The resulting fast tokenizer does not have a [PAD] token, and does not have any special tokens at all. ``` tokenizer = ByteLevelBPETokenizer() tokenizer.preprocessor = pre_tokenizers.BertPreTokenizer() tokenizer.normalizer = normalizers.BertNormalizer() tokenizer.train_from_iterator(docs, vocab_size=16_000, min_frequency=15, special_tokens = ["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]) tokenizer._tokenizer.post_processor = processors.BertProcessing( ("[SEP]", tokenizer.token_to_id("[SEP]")), ("[CLS]", tokenizer.token_to_id("[CLS]")), ) tokenizer.enable_truncation(max_length=256) tokenizer.enable_padding(pad_id=3, pad_token="[PAD]") fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer) ``` The result of printing the fast_tokenizer is: ``` PreTrainedTokenizerFast(name_or_path='', vocab_size=16000, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', special_tokens={}) ``` Which `model_max_len` and `special_tokens` are wrong in it. Also, there is no pad_token and pad_token_id in the fast_tokenizer object. (the warning for pad_token for example: `Using pad_token, but it is not set yet.`) Have I done anything wrong, or is this not supposed to happen? The versions of libraries I'm using: ``` ['tokenizers 0.10.3', 'transformers 4.10.0.dev0'] ```

Topic		Replies	Views
Tokenizer splits up pre-split tokens 🤗Tokenizers	9	6641	February 9, 2024
Pre-training a BERT model from scratch with custom tokenizer Intermediate	5	3097	January 11, 2022
Sentence splitting 🤗Tokenizers	7	31784	September 15, 2022
Bert pretrained tokenizer: how to preserve hyphened words? Beginners	0	311	April 6, 2022
Train a new tokenizer from scratch 🤗Transformers	4	1711	November 10, 2020

Tokenizer is splitting special token

Related topics