Why is BertNormalizer not exposed on the tokenizers library?

ndamulelonemakh · September 19, 2022, 12:03pm

Hi all

I am trying to use the BertNormalizer() to train a ByteLevelBPETokenizer(). But it seems this normaliser is not exposed looking at the source(see attached image). I am using tokenizers==0.12.1

Was this intentional or is it a feature that will be added going forward?

Topic		Replies	Views
Pre-training a BERT model from scratch with custom tokenizer Intermediate	5	2201	January 11, 2022
Where in the code does masking of tokens happen when pretraining BERT Beginners	5	5983	August 17, 2020
Using a BertWordPieceTokenizer trained from scratch from transformers 🤗Tokenizers	2	3664	March 26, 2021
Create DPR Tokenizer for non-Bert model Intermediate	1	278	September 7, 2021
Hey guys please could someone help me figure out the bert-restore-punctuation model Beginners	0	305	November 18, 2021

Why is BertNormalizer not exposed on the tokenizers library?

Related Topics