Getting ValueError when exporting model to ONNX using optimum

I’m trying to export my fine-tuned-bert-classifier torch model into ONNX format using optimum and then eventually want ro run it through a pipeline for sequence classification task.

Here is the code:

from optimum.onnxruntime import ORTModelForSequenceClassification, ORTOptimizer
from optimum.onnxruntime.configuration import OptimizationConfig
from optimum.pipelines import pipeline

# path_to_fine_tuned_model contains the path to the folder containing the pytorch_model.bin file
optimizer = ORTOptimizer.from_pretrained(path_to_fine_tuned_model, feature="sequence-classification") 
optimization_config = OptimizationConfig(optimization_level=2)

optimizer.export(
    onnx_model_path='../models/bert_model_opt.onnx',
    onnx_optimized_model_output_path='../models/bert_model_optimized.onnx',
    optimization_config=optimization_config,
)

However, this operation throws ValueError: Unable to generate dummy inputs for the model. Please provide a tokenizer or a preprocessor.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [35], in <cell line: 1>()
----> 1 optimizer.export(
      2     onnx_model_path='../models/bert_model_opt.onnx',
      3     onnx_optimized_model_output_path='../models/bert_model_optimized.onnx',
      4     optimization_config=optimization_config,
      5 )

File /opt/conda/envs/conda_ml/lib/python3.9/site-packages/optimum/onnxruntime/optimization.py:123, in ORTOptimizer.export(self, onnx_model_path, onnx_optimized_model_output_path, optimization_config, use_external_data_format)
    121 # Export the model if it has not already been exported to ONNX IR
    122 if not onnx_model_path.exists():
--> 123     export(self.preprocessor, self.model, self._onnx_config, self.opset, onnx_model_path)
    125 ORTConfigManager.check_supported_model_or_raise(self._model_type)
    126 num_heads = getattr(self.model.config, ORTConfigManager.get_num_heads_name(self._model_type))

File /opt/conda/envs/conda_ml/lib/python3.9/site-packages/transformers/onnx/convert.py:336, in export(preprocessor, model, config, opset, output, tokenizer, device)
    330         logger.warning(
    331             f"Unsupported PyTorch version for this model. Minimum required is {config.torch_onnx_minimum_version},"
    332             f" got: {torch_version}"
    333         )
    335 if is_torch_available() and issubclass(type(model), PreTrainedModel):
--> 336     return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer, device=device)
    337 elif is_tf_available() and issubclass(type(model), TFPreTrainedModel):
    338     return export_tensorflow(preprocessor, model, config, opset, output, tokenizer=tokenizer)

File /opt/conda/envs/conda_ml/lib/python3.9/site-packages/transformers/onnx/convert.py:143, in export_pytorch(preprocessor, model, config, opset, output, tokenizer, device)
    139         setattr(model.config, override_config_key, override_config_value)
    141 # Ensure inputs match
    142 # TODO: Check when exporting QA we provide "is_pair=True"
--> 143 model_inputs = config.generate_dummy_inputs(preprocessor, framework=TensorType.PYTORCH)
    144 device = torch.device(device)
    145 if device.type == "cuda" and torch.cuda.is_available():

File /opt/conda/envs/conda_ml/lib/python3.9/site-packages/transformers/onnx/config.py:347, in OnnxConfig.generate_dummy_inputs(self, preprocessor, batch_size, seq_length, num_choices, is_pair, framework, num_channels, image_width, image_height, tokenizer)
    345     return dict(preprocessor(images=dummy_input, return_tensors=framework))
    346 else:
--> 347     raise ValueError(
    348         "Unable to generate dummy inputs for the model. Please provide a tokenizer or a preprocessor."
    349     )

ValueError: Unable to generate dummy inputs for the model. Please provide a tokenizer or a preprocessor.

Any idea how to fix this? Thanks!

Hi @AmoghM! Thanks for using Optimum.

According to the traceback you provided, no tokenizer or preprocessor was found. In the folder containing your pytorch_model.bin file, is there a JSON file that defines your tokenizer/preprocessor?

1 Like

Thanks! The tokenizer files weren’t there an upon submitting it didn’t give the ValueError but now it is throwing this error message:

2022-08-17 21:30:22.151083668 [W:onnxruntime:, inference_session.cc:1546 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED and the NchwcTransformer enabled. The generated model may contain hardware specific optimizations, and should only be used in the same environment the model was optimized in.
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model
failed in shape inference <class 'AssertionError'>
failed in shape inference <class 'AssertionError'>

However, what is even more weird is that even after the AssertionError, the models were exported. And the latency on inference for initial onnx model, optimized onnx model and quantized onnx model is similar. I was expecting it to reduce.

Which version of Optimum, ONNX and ONNXRuntime do you use?

Here it is:

onnxruntime-gpu==1.12.1
optimum==1.3.0
onnx==1.12.0