I want to figure out some way to get faster results from a QA model. I did some tests on google cloud with different GPUs and got some results, those tests was made with different GPUs and same CPU using this code:
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
from transformers.pipelines import pipeline
tokenizer = AutoTokenizer.from_pretrained("deepset/bert-large-uncased-whole-word-masking-squad2")
model = AutoModelForQuestionAnswering.from_pretrained("deepset/bert-large-uncased-whole-word-masking-squad2")
nlp_qa = pipeline('question-answering', model=model, tokenizer=tokenizer)
X = nlp_qa(context = text, question=queryy, topk = 3, device = 0, max_answer_len = 50)
Where context is just a long string and the question a simple query, and I got those results:
TESTE 1: **********
4 vCPUs 15Gb RAM
NVIDIA TESLA P100X1
Tempo1: 1:45 min
Tempo2: 1:40 min
Tempo3: 1:45 min
***************
***************
TESTE 2: **********
4 vCPUs 15Gb RAM
NVIDIA TESLA V100X1
Tempo1: 1:58 min
Tempo2: 1:58 min
Tempo3: 1:55 min
***************
***************
TESTE 3: **********
4 vCPUs 15Gb RAM
NVIDIA TESLA K80X1
Tempo1: 2:06 min
Tempo2: 2:18 min
Tempo3: 2:20 min
***************
***************
TESTE 4: **********
4 vCPUs 15Gb RAM
NVIDIA TESLA T4X1
Tempo1: 1:45 min
Tempo2: 1:50 min
Tempo3: 1:50 min
***************
***************
TESTE 5: **********
4 vCPUs 15Gb RAM
NVIDIA NONE
Tempo1: 2:22 min
Tempo2: 1:57 min
Tempo3: 1:57 min
I want to know if I am using GPU wrong, or is it normal to get almost same results with and without GPU on this set?.
Is there anyway to get faster results?