I have converted the model (.h5) to tflite using:
converter = tf.lite.TFLiteConverter.from_keras_model(model) converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS] converter.optimizations = [tf.lite.Optimize.DEFAULT] tflite_model = converter.convert() open("/models/tflite_models/5th_Jan/distilbert_sms_60_5_jan.tflite", "wb").write(tflite_model)
but I also need tokenizer to run the model locally on the android app (independent of internet availability).
According to the articles on internet and question answered on stackoverflow How to tokenize input text in android studio to process in NLP model? we need json file of tokenizers to tokenize words in new inputs.
When I run the following code:
import json with open( 'android/word_dict.json' , 'w' ) as file: json.dump( tokenizer.word_index , file )
The following error comes:
AttributeError: 'DistilBertTokenizer' object has no attribute 'word_index
I am unable to find solution to use tokenizer of distilbert-base-uncased in android app. Any help will be appreciated. Thanks.