Hi! We would like to start a discussion about adding Intel OpenVINO backend in Transformers library.
If you have not heard about OpenVINO before, it’s a library which accelerates deep learning inference (not training, but inference of pretrained models) on Intel Architecture (CPU, GPU, VPU and others). The library is distributed in PyPI and developed in open source.
Currently, there is an issue here: Intel OpenVINO inference backend · Issue #13987 · huggingface/transformers · GitHub (for GitHub discussions) and the latest proposal here: Intel OpenVINO backend by dkurt · Pull Request #1 · dkurt/transformers · GitHub
Example (QA):
from transformers import AutoTokenizer, OVAutoModelForQuestionAnswering
tok = AutoTokenizer.from_pretrained("dkurt/bert-large-uncased-whole-word-masking-squad-int8-0001")
model = OVAutoModelForQuestionAnswering.from_pretrained("dkurt/bert-large-uncased-whole-word-masking-squad-int8-0001")
context = """
Soon her eye fell on a little glass box that
was lying under the table: she opened it, and
found in it a very small cake, on which the
words “EAT ME” were beautifully marked in
currants. “Well, I’ll eat it,” said Alice, “ and if
it makes me grow larger, I can reach the key ;
and if it makes me grow smaller, I can creep
under the door; so either way I’ll get into the
garden, and I don’t care which happens !”
"""
question = "Where Alice should go?"
input_ids = tok.encode(question + " " + tok.sep_token + " " + context, return_tensors="pt")
outputs = model(input_ids)
start_pos = outputs.start_logits.argmax()
end_pos = outputs.end_logits.argmax() + 1
answer_ids = input_ids[0, start_pos:end_pos]
answer = tok.convert_tokens_to_string(tok.convert_ids_to_tokens(answer_ids))
print("Question:", question)
print("Answer:", answer)