Hi.
I used the pre-trained weights of the DistilBERT model in order to train a text classification model with the SST-2 dataset. Now, when I am trying to export the trained model to SavedModel format, it fails. Here’s the full error trace.
Here’s the notebook for the training code. I use the following lines of code in order to export the trained model to SavedModel -
!mkdir DistilBERT
model.save("DistilBERT")
Hi @sayakpaul,
Where are you running this notebook, example, in Google Colab?
Hi @Karthik12. I am running it on Kaggle as a kernel.
Hi @sayakpaul, great to see you here 
this notebook might help. It exports TFDistilBertForQuestionAnswering in SavedModel format.
Thank you, @valhalla. I will give it a check and update
Looks like it was a silly incompatibility. With the GPU version of the same notebook (hyperparameters changed slightly), I was able to export the model to SavedModel. Does not look like a bug in transformers. Might be stemming from some incompatible Op within TensorFlow itself.
Looking at the stack trace, I guess it’s because TPU strategy can only access GCS file system, which is why it’s throwing UnimplementedError: File system scheme '[local]' not implemented.
Maybe pointing to gs:// address should allow export on TPU.
Just a guess, not super familiar with TF.
To be frank, huggingface support for Tensorflow is bare minimal. Its mostly focus on pytorch.
Yeah definitely. But it’s possible to export a model locally that’s been trained in a TPU runtime.
@sayakpaul Its not possible.