Zero-Shot Text Classification on a low-end CPU-only machine?

Beginners
1

I want to do zero-shot text classification either this model [1] (711 MB) or something similar:

Want to achieve high throughput in classification requests per second.
The classification will run on low-end hardware: some Hetzner [2] machine without GPU (Hetzner is great, reliable and cheap they just do not have GPU machines), something like this:
CCX13: Dedicated vCPU, 2 VCPU, 8 GB RAM
CX32: Shared vCPU, 4 VCPU, 8 GB RAM

Now there are multiple options for deploying and serving LLMs:
lmdeploy
text-generation-inference
TensorRT-LLM
vllm
There are more and more new frameworks for this. I am a bit lost.

Would you suggest the best option for deploying the above-listed model (No-GPU hardware)?

[1] MoritzLaurer/roberta-large-zeroshot-v2.0-c · Hugging Face
[2] https://www.hetzner.com/cloud/

P.S. I know about this post:

These recomendations - they’re great. But, uh, it’s been four years

1 Like
2

@joeddav @nielsr Any good updates in the last 4 years on zero-shot text classification, mainly in low-end environments without GPUs?

1 Like
3

You could check if Yake would be enough for your case, at least its resource demands are negligible compared to newer methods: GitHub - LIAAD/yake: Single-document unsupervised keyword extraction

1 Like
4

thanks @juhoinkinen !

as far as I understand: YAKE works for paragraph-long or document-long text.
and this zero-shot classification model in original text: it works well even with few words/sentence. This is what I need: sentence classification.