How to avoid PreTrainedTokenizerFast.decode to add space between tokens

[muzaffercky]

Hi, I trained a tokenizers. Tokens contain spaces as well. When I decode the decode method add space between tokens and it makes it wrong I need to avoid them. How to do that?

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("muzaffercky/kurdish-kurmanji-tokenizer")

test_text = """
Ez ê di vê gotarê da qala ên ku ez guhdar û temaşe dikim bikim
"""

tokens = tokenizer.tokenize(test_text)

print(f"Tokens: {tokens}")
# Tokens: ['\n', 'Ez ê ', 'di vê ', 'got', 'arê ', 'da ', 'qala ', 'ên ku ', 'ez ', 'guh', 'dar û ', 'temaşe ', 'dikim ', 'bikim', '\n']


ids = tokenizer.encode(test_text)
print(f"IDs: {ids}")

# IDs: [6, 6271, 1323, 452, 462, 396, 2409, 566, 654, 1204, 3278, 4543, 7880, 7595, 6]

text = tokenizer.decode(ids)

print(f"text: {text}")
# text: 
# Ez ê  di vê  got arê  da  qala  ên ku  ez  guh dar û  temaşe  dikim  bikim

As you can see it add extra space between tokens when decoding. I know I can make something like below but I am curious if transformer support something like this built-in

individual_tokens = [tokenizer.decode([id]) for id in ids]

"".join(individual_tokens)

[John6666]

Hmm… clean_up_tokenization_spaces?

What does the parameter 'clean_up_tokenization_spaces' do in the tokenizer.decode function?

As titled. The documentation is a bit vague. Whether or not to clean up the tokenization spaces.

[muzaffercky]

I do not understand what clean_up_tokenization_spaces does but it does not prevent adding space between tokens.

[John6666]

Hmm… need add_prefix_spaces or is PreTokenizerFast buggy?

Special token handling breaks idempotency of sentencepiece due to extra spaces

Set `add_prefix_space = False` for existing pre-trained tokenizers