Unable to load TF2-checkpoints into Huggingface

🤗Transformers
1

I am unable to import a pretrained BERT checkpoints trained with Tensorflow 2.

I get the error “AttributeError: ‘BertForPreTraining’ object has no attribute ‘shape’”. I have tried both the “transformers-cli convert” and the “.from_pretrained”, basically stopping at the same issue.

It also affects the official BERT models. I have no issues loading the official BERT-model trained for TF1.x, however trying to load the official TF2.0/Keras-model is giving me the same error.

This Colab Gist downloads the three official models hosted at gs://cloud-tpu-checkpoints and tries to load them into Huggingface:
https://colab.research.google.com/gist/peregilk/6b97384d3443ca9354a1898efd9819af/loading-tf-2-0-checkpoints-in-huggingface.ipynb

1 Like
2

Hey Pere,

Were you able to solve this issue? I have the a similar error but with bias. That is the error says “AttributeError: ‘BertForPreTraining’ object has no attribute ‘bias’”.

3

Long ago. From what I remembered I gave up on this. One of the reasons I switched to using PyTorch and Flax.