I want to fine-tune a bert model in Tensorflow (keras) and then visualise its attention weights in PyTorch (bertviz).
This documentation page https://huggingface.co/transformers/converting_tensorflow_models.html suggests that it might be possible to convert a tensorflow model to a pytorch model, but I don’t understand how to do it.
Firstly, what is transformers-cli? Is it a module? Do I need to import it? Where can I find out more about it? I’m using colab, so that includes transformers v 3.0.2 by default.
Secondly, when I click on the Examples links run_bert_extract_features.py, run_bert_classifier.py and run_bert_squad.py) I get a Github 404 page not found. Should they be there? Do I need to search within Github specificallly?
Thirdly, can the process be done within a colab jupyter notebook, or would I have to get a colab command line somehow? (Any hints on how would be appreciated!)
Which Tensorflow model are you starting from?
If it’s one from the library you probably don’t need to convert it.
If it’s an original checkpoint from another repository (typically from the original authors repository) then you should use the convertion command line utility. The transformers-cli is a command you can run from the command line when you have installed the transformers package.
Hi @thomwolf,
Initially I started from bert base on the tensorflow hub, https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/ , dropped the layers after NSP-Dense, added two layers, fine-tuned to my data, and saved the fine-tuned model with its weights. I did this using the keras-bert library in tensorflow. Ideally, I would like to bring the resulting fine-tuned tensorflow-keras model into transformers as a pytorch model, because I want to use Jesse Vig’s bertviz visualisation tool, which is using pytorch.
If that is not possible (and it doesn’t look promising), then I would like to use a transformers bert-base-uncased TFBert model, amend and tune it using tensorflow, then convert the result to pytorch.
Would you expect either of these to be possible? If so, do I need to use the cli, or just the from_tf=True parameter?
Hmm then if the structure of the model is itself different for the original structure you will need to write a full model class in PyTorch I’m afraid
thanks for replying. I’ve started again just using pytorch with transformers.