MNLI Inference on a fine-tuned model from hub

Beginners
1

Hello,
I tried to run various models such as

  • huggingface/distilbert-base-uncased-finetuned-mnli
  • microsoft/deberta-v2-xxlarge-mnli
  • roberta-large-mnli
  • squeezebert/squeezebert-mnli

I can see the weights are loaded but the accuracy I get is about 7%.
I use “transformers_version”: “4.3.2”, and the following arguments:

python run_glue.py --model_name_or_path roberta-large-mnli --task_name mnli --do_eval --max_seq_length 128 --output_dir /tmp/mnli/

Any help would be appreciated,
Thank you

2

Hi Ali93H,

I don’t understand what you are trying to do. Are you wanting to do further fine-tuning (in which case you might want DistilBertForMaskedLM) or to classify your texts for example sentiment analysis (in which case you might want DistilBertForSequenceClassification).

The transformers docs are here Quick tour — transformers 4.3.0 documentation ,Summary of the models — transformers 4.3.0 documentation , DistilBERT — transformers 4.3.0 documentation

3

Thanks for your reply. I’m trying to do an evaluation on an already fine-tuned model (reproduce the accuracy metrics).
If I try this with mrpc or stsb (with fine-tuned models, of course) it works just fine, but I don’t know what is the issue with MNLI.

4

Here is the source of the issue and a solution: [run_glue] Add MNLI compatible mode by JetRunner · Pull Request #10203 · huggingface/transformers · GitHub

1 Like