i try to export T5 model to ONNX but when a try to predict something it show a error:
sess = ort.InferenceSession(‘onnx\model.onnx’)
inputs = tokenizer(“something in here”, return_tensors=“np”)
outputs = sess.run( None,inputs )
ValueError: Model requires 4 inputs. Input Feed contains 2
Maybe @lewtun will have an idea?
Hey @Slcreator the number of inputs you need for inference depends on the type of head you chose to export the model with (i.e. the feature
arg in the transformers.onnx
package).
For example, here’s how you can figure out which inputs are needed for say the default
feature of a T5 checkpoint:
from transformers.models.t5 import T5Config, T5OnnxConfig
config = T5Config.from_pretrained("t5-small")
onnx_config = T5OnnxConfig(config=config, task="default")
# returns
# {'input_ids': {0: 'batch', 1: 'encoder_sequence'},
# 'attention_mask': {0: 'batch', 1: 'encoder_sequence'},
# 'decoder_input_ids': {0: 'batch', 1: 'decoder_sequence'},
# 'decoder_attention_mask': {0: 'batch', 1: 'decoder_sequence'}}
onnx_config.inputs
You can change the task
arg to match the type of head you’re exporting and that will tell you what the model expects as inputs.
Btw we’ve recently added a pipeline()
function to the optimum
library, which makes it quite simple to run inference on ONNX Runtime. There’s a guide here if you’re interested: Optimum pipelines for inference
1 Like
thanks very much, i’m going to check it again
1 Like
Hello, I am trying to use Bigbirdpegasus and getting the same error. Model requires 4 inputs but how can we provide decoder attention and mask beforehand? isn’t that supposed to be generated after we get output from the model?
Can you please share the code to export bigbird for summarization?
Here is the code I am using:
from pathlib import Path
from transformers.onnx import export
from transformers import AutoTokenizer, AutoModel
onnx_path = Path(“model.onnx”)
model_ckpt = “google/bigbird-pegasus-large-pubmed”
base_model = AutoModel.from_pretrained(model_ckpt)
tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, onnx_config.default_onnx_opset, onnx_path)
tokenizer = AutoTokenizer.from_pretrained(“google/bigbird-pegasus-large-pubmed”)
session = InferenceSession(“model.onnx”)
inputs = tokenizer(“Summarize this text”, return_tensors=“np”)
inputs = {k: v.astype(np.int64) for k, v in inputs.items()}
outputs = session.run(output_names=[“last_hidden_state”], input_feed=inputs)