Is there any way to load a google released ckpt in https://github.com/google-research/bert (because some settings are not provided in hf transformers) using tf 2+ ?
I’ve found docs on loading those models and converting them to pytorch models in here:
https://huggingface.co/transformers/converting_tensorflow_models.html
However, it only works with pytorch.
Have you looked at using the keras-bert library (instead of using huggingface transformers library)?
keras-bert works with TF2. I used it to load a bert model from the tensorflow hub into colab. I copied it onto my google drive first, and loaded from that copy, but I believe it is also possible to load directly using
!wget -q https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip
!unzip -o uncased_L-12_H-768_A-12.zip
So far as I can tell, models loaded with keras-bert are not compatible with transformers. I tried to get my fine-tuned model into transformers format (because I wanted to use Jesse Vig’s visualisations), but I never got it to work.
I believe there is something about the way the transformers code is written that means you can’t use non-transformers models in place of transformers models, or at least not without some serious poking of the python. However, I could be mistaken.
Hello again, it looks like I was mistaken.
The suggestion to use keras-bert instead still stands, but I’ve had another look at the transformers documentation and it certainly looks like it should be possible to use the google models within transformers. See eg this issue https://github.com/huggingface/transformers/issues/3070.
Are you explicitly loading the google model’s config file?
What happens if you try using the first part of the pytorch conversion script to read the google model in?
Thanks a lot for your comments!
Yes I’m explicitly loading the config file, which I believe is almost the same in both tf and pytorch.
I’ve tried to use the pytorch conversion script to load the model in pytorch and then convert the pytorch model to tensorflow model, but I can’t find any scripts to convert the pytorch model to tensorflow one…
However, I’ve found a workaround, which is a temper fix: Since the may problem is the mismatch of tensor names, I load the google checkpoint file and rename tensors to match those in transformers. Fortunately the mismatches are easily located and only need sightly modification.