Import distilbert-base-uncased tokenizer to an android app along with the tflite model

🤗Tokenizers
1

I have converted the model (.h5) to tflite using:

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, 
                                       tf.lite.OpsSet.SELECT_TF_OPS]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
open("/models/tflite_models/5th_Jan/distilbert_sms_60_5_jan.tflite", "wb").write(tflite_model)

but I also need tokenizer to run the model locally on the android app (independent of internet availability).

According to the articles on internet and question answered on stackoverflow How to tokenize input text in android studio to process in NLP model? we need json file of tokenizers to tokenize words in new inputs.

When I run the following code:

import json

with open( 'android/word_dict.json' , 'w' ) as file:
    json.dump( tokenizer.word_index , file )

The following error comes:

AttributeError: 'DistilBertTokenizer' object has no attribute 'word_index

I am unable to find solution to use tokenizer of distilbert-base-uncased in android app. Any help will be appreciated. Thanks.

2

excuse me did you solve it ?

3

word_index is an attribute of Keras’ own tokenizer, not Hugging Face. See docs here.

4

excuse me . what is the equivalent manner in hugging face ?