Text generation. Stream output

🤗Transformers
1

Hi, I want to use text generation and stream the output similar to ChatGPT. How to do that?

1 Like
2

I made a streaming generation service for Hugging Face transformers that is fully compatible with the OpenAI API: https://github.com/hyperonym/basaran

Hope it meets your needs.

2 Likes
3

I checked out basaran lasted week and its great! Loved the fact that it worked streaming both with t5 and gpt like models! Also the playground was really nice!