I’m hitting what seems to me to be an odd limit on the number of characters a Word Piece tokenizer will process before returning [UNK]. I’m working on a project which uses long strings of generated characters that I’m presenting to BERT as a long, ‘strange-looking’ word. Any word less than 100 chara…

Help with Tokenizer Word Length Limit

smagennis July 16, 2023, 6:50pm 3

Brilliant! Thanks mormart!

Topic		Replies	Views
Changing Tokenizer's max_length gets weird result Beginners	2	437	May 17, 2022
Is 512 token in bert, token or character level? Beginners	3	9353	April 4, 2022
Number of words Beginners	6	9051	February 22, 2021
How to ensure that tokenizers never truncate partial words? 🤗Tokenizers	2	1808	January 24, 2022
WordPiece tokenizer doesn't work for long sequences 🤗Tokenizers	1	397	March 28, 2024