Unable to load TF2-checkpoints into Huggingface

I am unable to import a pretrained BERT checkpoints trained with Tensorflow 2.

I get the error “AttributeError: ‘BertForPreTraining’ object has no attribute ‘shape’”. I have tried both the “transformers-cli convert” and the “.from_pretrained”, basically stopping at the same issue.

It also affects the official BERT models. I have no issues loading the official BERT-model trained for TF1.x, however trying to load the official TF2.0/Keras-model is giving me the same error.

This Colab Gist downloads the three official models hosted at gs://cloud-tpu-checkpoints and tries to load them into Huggingface:

1 Like

Hey Pere,

Were you able to solve this issue? I have the a similar error but with bias. That is the error says “AttributeError: ‘BertForPreTraining’ object has no attribute ‘bias’”.

Long ago. From what I remembered I gave up on this. One of the reasons I switched to using PyTorch and Flax.