Hey there!
While in my context to tokenize, there is no empty string, e.g. there is no
'' in the real words.
However, when tokenizing, the tokenizer creates token (token id 6 with the LayoutXLM tokenizer), which, upon decoding, generate this very empty string.
What am I missing?
E.g. the orginal word in “No.” →
tokenizing → decoding → to “no” “.” “”.
See here the data example, where I output the words and corresponding token:
token idx =41
token dec = Zürich
<class 'str'>
token idx =42
token dec =
<class 'str'>
this token is decoded to an empty str:
6
token idx =43
token dec = ZH
<class 'str'>
word idx: 101
word: Zürich
char: Z
char: ü
char: r
char: i
char: c
char: h
word idx: 102
word: ZH
char: Z
char: H
What am I missing? Help very much appreciated!
Best,
Seb