Optimum Transformers
Accelerated NLP pipelines for fast inference 
on CPU and GPU. Built with 
Transformers, Optimum and ONNX runtime.
Disclaimer
This project is my inspiration of Huggingface Infinity. And first step done by Suraj Patil.
@huggingface’s pipeline API is awesome!
, right? And onnxruntime is super fast !
– Tweet by Suraj Patil
It was under this slogan that I started doing this project!
And the main goal was to show myself to get into @huggingface team
How to use
Quick start
The usage is exactly the same as original pipelines, except minor improves:
from optimum_transformers import pipeline
pipe = pipeline("text-classification", use_onnx=True, optimize=True)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]
use_onnx- converts default model to ONNX graphoptimize- optimizes converted ONNX graph with Optimum
Optimum config
Read Optimum documentation for more details
from optimum_transformers import pipeline
from optimum.onnxruntime import ORTConfig
ort_config = ORTConfig(quantization_approach="dynamic")
pipe = pipeline("text-classification", use_onnx=True, optimize=True, ort_config=ort_config)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]
Benchmark
With notebook
You can benchmark pipelines easier with benchmark_pipelines notebook.
With own script
from optimum_transformers import Benchmark
task = "sentiment-analysis"
model_name = "philschmid/MiniLM-L6-H384-uncased-sst2"
num_tests = 100
benchmark = Benchmark(task, model_name)
results = benchmark(num_tests, plot=True)
Results
Note: These results were collected on my local machine. So if you have high performance machine to benchmark, please contact me
sentiment-analysis
Almost the same as in Inifinity launch video
AWS VM: g4dn.xlarge
GPU: NVIDIA T4
128 tokens
2.6 ms
zero-shot-classification
With typeform/distilbert-base-uncased-mnli
token-classification
More results are available in project repository: GitHub.
