MNLI Inference on a fine-tuned model from hub

I tried to run various models such as

  • huggingface/distilbert-base-uncased-finetuned-mnli
  • microsoft/deberta-v2-xxlarge-mnli
  • roberta-large-mnli
  • squeezebert/squeezebert-mnli

I can see the weights are loaded but the accuracy I get is about 7%.
I use “transformers_version”: “4.3.2”, and the following arguments:

python --model_name_or_path roberta-large-mnli --task_name mnli --do_eval --max_seq_length 128 --output_dir /tmp/mnli/

Any help would be appreciated,
Thank you

Hi Ali93H,

I don’t understand what you are trying to do. Are you wanting to do further fine-tuning (in which case you might want DistilBertForMaskedLM) or to classify your texts for example sentiment analysis (in which case you might want DistilBertForSequenceClassification).

The transformers docs are here Quick tour — transformers 4.3.0 documentation ,Summary of the models — transformers 4.3.0 documentation , DistilBERT — transformers 4.3.0 documentation

Thanks for your reply. I’m trying to do an evaluation on an already fine-tuned model (reproduce the accuracy metrics).
If I try this with mrpc or stsb (with fine-tuned models, of course) it works just fine, but I don’t know what is the issue with MNLI.

Here is the source of the issue and a solution: [run_glue] Add MNLI compatible mode by JetRunner · Pull Request #10203 · huggingface/transformers · GitHub

1 Like