Best practice for upgrading models?

Models
1

Hi everybody. I’m looking for best practices for upgrading models which have been trained with Transformers. We have a model which is trained with Transformers 4.0.x, which breaks with the latest version (4.1.1). We had the same issue with models trained with Transformers 3 when Transformers 4 came out.

Retraining models every time the Transformers package is updated is expensive and the new model might give different output. Pinning the version is of course possible, but we would prefer to keep the models up-to-date.

Are there other options than fully retraining the model or pinning the version that we could consider? We save our models and tokenizers with the save_pretrained method.

2

How does your model “breaks”? Can you give more details?

3

Normally you should not need to retrain models indeed.

4

When I try to load the model, which was trained with an older version of Transformers, now in my env with Transformers 4.1, I get the following error:
'BertEmbeddings' object has no attribute 'position_embedding_type'

I had similar issues when we upgraded from Transformers 3 to 4, attributes that were missing.

5

Hi @RobinvdM, sorry you’re experiencing this issue, it should not happen. Do you mind sharing the code you used to reload the model, as well as the stacktrace?

6

This is the code used to reload the model.

        if torch.cuda.is_available():
            device = torch.device("cuda")
        else:
            device = torch.device("cpu")
        model = torch.load(model_path, map_location=device)
        if torch.cuda.is_available():
            model.cuda()
        model.eval()
        input_ids = dataset[0].to(device)
        input_masks = dataset[1].to(device)
        with torch.no_grad():
            logits = model(
                input_ids,
                token_type_ids=None,
                attention_mask=input_masks,
                return_dict=False,
                labels=None
            )[0]

The error occurs after the model() call. Stack trace:


/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:722: in _call_impl
    result = self.forward(*input, **kwargs)
/usr/local/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:1375: in forward
    return_dict=return_dict,
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:722: in _call_impl
    result = self.forward(*input, **kwargs)
/usr/local/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:862: in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:722: in _call_impl
    result = self.forward(*input, **kwargs)
/usr/local/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:202: in forward
    if self.position_embedding_type == "absolute":
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = BertEmbeddings(
  (word_embeddings): Embedding(30000, 768, padding_idx=3)
  (position_embeddings): Embedding(512, 768)... 768)
  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
  (dropout): Dropout(p=0.1, inplace=False)
)
name = 'position_embedding_type'

    def __getattr__(self, name: str) -> Union[Tensor, 'Module']:
        if '_parameters' in self.__dict__:
            _parameters = self.__dict__['_parameters']
            if name in _parameters:
                return _parameters[name]
        if '_buffers' in self.__dict__:
            _buffers = self.__dict__['_buffers']
            if name in _buffers:
                return _buffers[name]
        if '_modules' in self.__dict__:
            modules = self.__dict__['_modules']
            if name in modules:
                return modules[name]
        raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
>           type(self).__name__, name))
E       torch.nn.modules.module.ModuleAttributeError: 'BertEmbeddings' object has no attribute 'position_embedding_type'

Thank you for your help

7

In your code I’m not seeing how your model is built, which is probably where the issue lies. Is it a model you created yourself, which does not inherit from PretrainedModel, hence why you’re using torch.load instead of from_pretrained?

If so, could you show me your model creation so that I can try and replicate on my side? Thank you.

8

Also @RobinvdM you can consider uploading your model to the Huggingface.co model hub if you want us to take a look.

You can upload it privately if needed.

9

Does that mean that you (as maintainer of the storage) have access to users’ private models? I guess so, but I am not sure if that is desired for some people who expect real private storage for their models. (If this is disclaimed somewhere already when taking a subscription, that is perfect. It should be clear to subscribers what “private” means.)