Improving decoding speed by onnx conversion model

outstanding · November 17, 2021, 7:52am

After I use the uer py training model to convert the transformers model, the decoding speed is not very fast. I want to improve the speed by converting the model onnx.

python -m transformers.onnx --model=bert-base-cased onnx/bert-base-cased/

Some problems were encountered using the converted model.
According to the official documentation, there is no generation method.

I checked the case list of gpt2 conversion onnx. The gpt2 model can be decoded normally and the speed has been improved.

github.com

microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/notebooks/Inference_GPT2_with_OnnxRuntime_on_CPU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference PyTorch GPT2 Model with ONNX Runtime on CPU\n",
    "\n",
    "In this tutorial, you'll be introduced to how to load a GPT2 model from PyTorch, convert it to ONNX, and inference it using ONNX Runtime using IO Binding. Note that past state is used to get better performance."
   ]
  },
  {

This file has been truncated. show original

An exception occurred after replacing with the converted Chinese model.

Hope to get help here, thank you!

Topic		Replies	Views
Using onnx for text-generation with GPT-2 🤗Transformers	4	2900	February 3, 2023
Gpt2 inference with onnx and quantize Beginners	6	3393	February 3, 2021
Boost inference speed of T5 models up to 5X & reduce the model size by 3X 🤗Transformers	2	4739	June 8, 2023
When exporting seq2seq models with ONNX, why do we need both decoder_with_past_model.onnx and decoder_model.onnx? 🤗Optimum	12	2715	March 7, 2024
How to convert mT5 and ByT5 to ONNX format? 🤗Transformers	4	1723	December 22, 2021

Improving decoding speed by onnx conversion model

Related Topics