RoBERTa large: HF vs. FAIRseq

NLPYeahUKnowMe · May 8, 2024, 8:17pm

Hi,
I’m trying to understand what the difference is between the model

…and the version available for download here:

facebookresearch/fairseq/blob/main/fairseq/models/roberta/model.py#L39


      
          
          logger = logging.getLogger(__name__)
          
          
          @register_model("roberta")
          class RobertaModel(FairseqEncoderModel):
              @classmethod
              def hub_models(cls):
                  return {
                      "roberta.base": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.base.tar.gz",
                      "roberta.large": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.large.tar.gz",
                      "roberta.large.mnli": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.large.mnli.tar.gz",
                      "roberta.large.wsc": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.large.wsc.tar.gz",
                  }
          
              def __init__(self, args, encoder):
                  super().__init__(encoder)
                  self.args = args
          
                  # We follow BERT's random weight initialization
                  self.apply(init_bert_params)

For example, the HF version is 1.42GB as safe tensors, but the FAIRseq version is only 678 MB.

Happy to read documentation if this is described somewhere.
Thanks in advance,
Dan

nielsr · May 9, 2024, 9:35am

Hi,

That’s a good question. I would recommend inspecting the conversion script: transformers/src/transformers/models/roberta/convert_roberta_original_pytorch_checkpoint_to_pytorch.py at main · huggingface/transformers · GitHub.

Topic		Replies	Views
Discrepancy in Model Inference: Local vs. Hugging Face Model Hub 🤗Transformers	1	773	December 27, 2023
How do I fine-tune roberta-large for text classification Beginners	7	3334	December 17, 2021
How can I share a pytorch saved model on huggingFace hub Beginners	0	656	May 4, 2022
Loading pretrained SentencePiece tokenizer from Fairseq 🤗Tokenizers	5	6108	October 21, 2020
Difference between roberta and bert for pretraining Models	0	529	July 15, 2023

RoBERTa large: HF vs. FAIRseq

Related topics