Inference with Finetuned BERT Model converted to ONNX does not output probabilities

Intermediate
1

Environment info

  • transformers version: 3.5.1
  • Platform: Linux-4.14.203-116.332.amzn1.x86_64-x86_64-with-glibc2.10
  • Python version: 3.7.6
  • PyTorch version (GPU?): 1.7.0 (True)
  • Tensorflow version (GPU?): 2.3.1 (True)
  • Using GPU in script?: No
  • Using distributed or parallel set-up in script?: No

Information

Model I am using (Bert, XLNet …): Bert

The problem arises when using:

  • my own modified scripts: (give details below)

The tasks I am working on is:

  • my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

  1. Trained HuggingFace Transformers model BertForSequenceClassification on custom dataset with PyTorch backend.
  2. Used provided convert_graph_to_onnx.py script to convert model (from saved checkpoint) to ONNX format.
  3. Loaded the model with ONNXRuntime
  4. Instantiated BertTokenizer.from_pretrained(‘bert-based-uncased’) and fed in various input text to encode_plus method.
  5. Fed outputs of this to the ONNXRuntime session.

Expected behavior

The expected behavior is that the output of sess.run on the aforementioned inputs should output an array of dimension (1, 100) (corresponding to 100 classes) with each value between 0 and 1, with all entries summing to 1. We get the correct dimension, however, we get values between about -3.04 and 7.14 (unsure what these values refer to).