Hi, I want to use text generation and stream the output similar to ChatGPT. How to do that?
1 Like
I made a streaming generation service for Hugging Face transformers that is fully compatible with the OpenAI API: https://github.com/hyperonym/basaran
Hope it meets your needs.
2 Likes
I checked out basaran lasted week and its great! Loved the fact that it worked streaming both with t5 and gpt like models! Also the playground was really nice!