Missing `model_type` key in config.json of TinyBERT

🤗Transformers
1

Hello,

I’m trying to use one of the TinyBERT models produced by HUAWEI (link) and it seems there is a field missing in the config.json file:

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("huawei-noah/TinyBERT_General_4L_312D")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lewtun/git/transformers/src/transformers/models/auto/tokenization_auto.py", line 345, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/Users/lewtun/git/transformers/src/transformers/models/auto/configuration_auto.py", line 360, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in huawei-noah/TinyBERT_General_4L_312D. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: retribert, mt5, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, mpnet, bart, blenderbot, reformer, longformer, roberta, deberta, flaubert, fsmt, squeezebert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag, tapas

Looking at the config.json file (link) it seems like it should be an easy enough fix to add something like "model_type": "tinybert" so my question is how does one go about patching a fix in a community provided model? Do I raise an issue on the :hugs: Transformers repo or somewhere else?

1 Like
2

Oh that field is missing indeed. From what I know though, it can’t be filled with “tinybert” as it needs to be a model type implemented in the library, so I think it should be “bert”. As for adding it, I think it needs to be done on our side (in the future we’ll have the ability to open PRs on model repos, which would help in that case!)

Looking at that config, there are a few things that are a bit weird, so it may be that they have an internal class for those models not compatible with Transformers.

2 Likes
3

Thanks for the info Sylvain!

I agree that the config is a bit weird, so I’ve posted a question on the FastFormers repo (link) whose experiments I am trying to reproduce and who somehow found a way to make TinyBERT work with :hugs: Transformers.

Failing that, I’ll ask the TinyBERT authors directly and report back here :slight_smile:

4

I’m experiencing the same issue with the BlenderBot models.
After executing config.save_pretrained(model_dir), the model_type:'blenderbot' attribute is missing from the config.json file

5

@lewtun - Regarding TinyBERT, have you checked Albert joint model from GitHub - legacyai/tf-transformers: State of the art faster Natural Language Processing in Tensorflow 2.0 . .
Glue score on Albert base 14M and 6 layer seems to have 81, which is better than Tinybert, Mobilebert, distillbert, which has 60M parameter.