Application of TFBertTokenizer

🤗Tokenizers
1

I wish to use the TFBertTokenizer as a layer in my custom model where there is:

  • Input Layer - Receiving String
  • Tokenizer to process string tensor
  • Bert model
  • Other layers of my custom layer

The model is for classification problem and the intentions are to have a model that has both preprocessing as part of the model. I’m thinking this is possible since TFBertTokenizer is an in-graph tokenizer.

Please help.