Accelerate: 'RobertaModel' object has no attribute 'roberta'

Intermediate
1

I am trying to load a RobertaModel using Accelerate, following the example in the documentation.

This is my code:

        with init_empty_weights():
            model = RobertaModel.from_pretrained(model_name_or_path)
        model = load_checkpoint_and_dispatch(
            model, checkpoint=checkpoint, device_map="auto"
        )

However, this results in an error:

AttributeError: 'RobertaModel' object has no attribute 'roberta'

I have tried with various RobertaModels as well as with XmodModel, whereas doing the same with a AutoModelForMaskedLM (for instance) works. Is RobertaModel not supported by Accelerate?

Is there a list of supported model types somewhere? I could not find any in the documentation.

2

I suppose the given example was wrong for my use case. This implementation seems to work for me with all model types:

model = RobertaModel.from_pretrained(...)
accelerator = Accelerator()
accelerator.prepare(model)