Error while saving and loading a Bert model

🀗Transformers
1

Hello ,

I am very new to transformers and this is a model I built using the transformers library for text classification.

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')

model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')

optimizerr = tf.keras.optimizers.Adam(learning_rate=5e-5)
losss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=optimizerr,loss=losss,metrics=['accuracy'])  

model.fit(train_dataset.shuffle(train.count()).batch(16), epochs=1,batch_size=16)

Now My goal is to save this trained model and load it at a later point of time to make prediction.

To save the model I tried :

path = '/Shared/saved_models'
model.save_pretrained(path, saved_model=True)

I am getting the below warnings when I do this :

WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.dropout.Dropout object at 0x7f51b81941c0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.dropout.Dropout object at 0x7f51b828cf10>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.dropout.Dropout object at 0x7f51b81086a0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.dropout.Dropout object at 0x7f51b830ed90>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.dropout.Dropout object at 0x7f51b83194f0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.dropout.Dropout object at 0x7f51b8322c10>, because it is not built.
WARNING:absl:Found untraced functions such as embeddings_layer_call_fn, embeddings_layer_call_and_return_conditional_losses, transformer_layer_call_fn, transformer_layer_call_and_return_conditional_losses, LayerNorm_layer_call_fn while saving (showing 5 of 164). These functions will not be directly callable after loading.

Now to load the model to make my predictions , I am doing :

new_model = AutoModelForSequenceClassification.from_pretrained('/Shared/saved_models',from_tf=True)

This gives me another warning :

All TF 2.0 model weights were used when initializing DistilBertForSequenceClassification.
All the weights of DistilBertForSequenceClassification were initialized from the TF 2.0 model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use DistilBertForSequenceClassification for predictions without further training.

But I am not able to make predictions using this ‘new_model’ .
I am not able to fully understand what is happening and how I can sucesfully save and load my model. Any help is highly appreciated. Thank you.

1 Like