Hey @charly, here’s a previous thread about that. The main tricks are going to be:
- Use one of these distilled models which are smaller and faster but with similar results
- Run with the ONNX Runtime. One way you can do this is with this project created by @valhalla before he joined Hugging Face
- If you have long sequences you’re classifying, you can try truncating to just part of the sequence. That’ll give you a speedup but you’ll have to evaluate how it impacts your performance.
- If you have a large # of candidate labels, try to come up with a heuristic or use a super lightweight classifier to identify the most likely candidates, and then just feed in those more likely candidates rather than all of them.
Btw if it’s public would you mind linking to your streamlit app? It’s always fun to see the ways that people are using it 