Supporting ONNX optimized models

🤗Transformers
#1

In a lot of cases, ONNX optimized models offer much better performance benefits as compared to using PyTorch models.
This performance boost coupled with the pipelines offered by HuggingFace are a really great combo for delivering a great experience both in terms of inference speed and model performance.

Right now, it’s possible to use ONNX models with a little bit of modification to the pipeline.py code. On my tests on the QuestionAnsweringPipeline with SQuADv2 dataset, I see performance improvements of 1.5-2x with models like bert-large-uncased-whole-word-masking-finetuned-squad and distilbert-base-cased-distilled-squad

I’m wondering if this is something that the devs/community considers worthwhile for supporting directly in the transformers repo. I’m doing some work on this for my own project, but it’s mostly a hacky impl. at this point.
If there’s greater interest in this then I could try to integrate ONNX support for inference more fully in the code base.

Would love to hear everyone’s thoughts.

Thanks

3 Likes
[RFC] Transformers Pipeline v2
#2

I know @mfuntowicz is working on this so he may have more input.

2 Likes
#3

We would love to see onnx support in the pipelines, it would make serving models much more scalable.
Actually we have never worked with onnx, but the statistics look pretty good.

What do you think about automatically generating onnx models next to TF and pytorch models, so over the time any uploaded model on HF hub would be available as onnx model too.

We would love to help you there, but need to read more into onnx runtime.

1 Like
#4

Very cool pipelines implementation

#5

Hi there,

Are there any plans for Huggingface to distribute pre-trained models as ONNX files? My use case is embedded pre-trained model inference.

I don’t necessarily need the raw pytorch or tensorflow model. The default ONNX quantized export would be enough for me.

I can do that myself using a combination of the huggingface download utils and the ONNX conversion script. Ideally, I would get the ONNX file directly from :hugs:

[ADDITION] It would probably not be as simple as “get the ONNX file” if we need a plug-and-play experience. Happy to brainstorm on the right format!

Cheers,

Alex Combessie

1 Like
#6

Hi @dataiku, give this a try https://github.com/patil-suraj/onnx_transformers

Note: this is a utility project to allow easy inference, which creates the grapth on the fly.

Pinging @julien-c to see if it’s possible to host onnx files on hub.

1 Like
#7

Hi @valhalla,

Just had a look. I am exactly looking for that kind of simplicity: nlp = pipeline("sentiment-analysis", onnx=True) :slight_smile:

Good prototype. The main caveats I can think of are:
(i) ONNX conversion is done on-device - I’d rather pull pre-computed ONNX files from the model hub
(ii) some level duplication of huggingface source code
(iii) possible improvements in the tokenizer choices to use fast versions when available

I would really like to have this option native in huggingface so I can use it in production applications when inference speed matters a lot.

Cheers,

Alex

#8

i) we could pull if they are available
ii) Duplicated to provide the same API
iii) Easy to add. Will add it.

HF is considering this in the pipeline V2
see [RFC] Transformers Pipeline v2

#9

Thanks for pointing it out to [RFC] Transformers Pipeline v2. That’s good to know.

For (ii) is it possible to leverage Python class inheritance features to only add/override the methods which are necessary for ONNX?

#10

definitely possible! Copied everything so that I can have full control

#11

We do have a few model in onnx on the hub, and we support it as a searchable filetype: https://huggingface.co/models?filter=onnx

(cc @mfuntowicz)

We’ll look into auto-converting models to onnx in the future. For now, feel free to ask model authors (or us) to upload converted files manually

2 Likes
#12

If we upload onnx models, will they appear in onnx filter ?

#13

For now there is only two models at https://huggingface.co/models?filter=onnx.

And I agree with @bdalal, it’s “quite easy” to convert a model to an onnx checkpoint and use it in let’s say NodeJs which don’t have a mature ecosystem for Deeplearning.

I would convert them it if I need to fintune a model but for hub models it would be more convenient to download them directly (if converted automatically). I almost envisaged to convert the models I use and host them for my team to use…

Also a model.save(args, onnx=True) to save a onnx file locally (by calling convert_graph_to_onnx) as well would be really useful !

So here is the 2cents of a really happy user :slight_smile:

Thanks for this wonderfull tool !
Have a great day