Pipeline very slow

🤗Transformers
1

I just trained a BertForSequenceClassification classifier but come on problems when trying to predict.

When I use the predict method of trainer on encodings I precomputed, I’m able to obtain predictions for ~350 samples from test set in less than 20 seconds.

trainer.predict(test_encodings)

However, when I load the model from storage and use a pipeline, the code runs for more than 10 mins, even adding batching doesn’t seem to help.

classifier = pipeline("text-classification", model=model,tokenizer=tokenizer,max_length=512, truncation=True)
classifier(X_test.Text.to_list(),batch_size=10)

What can explain this difference ?

5 Likes
2

Can’t offer any solutions, but I’m experiencing a similar problem training a model in the HuggingFace hub and trying to load it with the from_pretrained method to make predictions locally. I’m averaging 6s / observation - not something I can ever use in production.

This SO post reports basically the same issue with from_pretrained…