SavedModel export for DistilBERT is failing

🤗Transformers
1

Hi.

I used the pre-trained weights of the DistilBERT model in order to train a text classification model with the SST-2 dataset. Now, when I am trying to export the trained model to SavedModel format, it fails. Here’s the full error trace.

Here’s the notebook for the training code. I use the following lines of code in order to export the trained model to SavedModel -

!mkdir DistilBERT
model.save("DistilBERT")
2

Hi @sayakpaul,

Where are you running this notebook, example, in Google Colab?

3

Hi @Karthik12. I am running it on Kaggle as a kernel.

4

Hi @sayakpaul, great to see you here :slight_smile:
this notebook might help. It exports TFDistilBertForQuestionAnswering in SavedModel format.

5

Thank you, @valhalla. I will give it a check and update :slight_smile:

6

Looks like it was a silly incompatibility. With the GPU version of the same notebook (hyperparameters changed slightly), I was able to export the model to SavedModel. Does not look like a bug in transformers. Might be stemming from some incompatible Op within TensorFlow itself.

7

Looking at the stack trace, I guess it’s because TPU strategy can only access GCS file system, which is why it’s throwing UnimplementedError: File system scheme '[local]' not implemented.
Maybe pointing to gs:// address should allow export on TPU.
Just a guess, not super familiar with TF.

8

To be frank, huggingface support for Tensorflow is bare minimal. Its mostly focus on pytorch.

9

Yeah definitely. But it’s possible to export a model locally that’s been trained in a TPU runtime.

1 Like
10

@sayakpaul Its not possible.