Sorry if I missed this in the documentation.
Are user requests processed in parallel or sequential?
Many thanks in advance for your time.
Here are some key points to consider:
- Firstly, the request capacity will depend on the hardware resources behind the server.
- Secondly, the processing of requests can be influenced by the chosen SDK.
- If you’re using Gradio, it utilizes uvicorn and FastAPI as the web server, which can queue requests for your prediction function. Gradio also provides a way to batch requests for better performance: Setting Up A Demo For Maximum Performance
- When using Streamlit, you can learn more about how it handles requests here: Does streamlit is running on a single-threaded development server by default or not? - #2 by tim - 🚀 Deployment - Streamlit
- If you’re employing the Docker SDK, it’s up to you to set up a web server and manage multiple requests.
In summary, the processing of user requests will depend on the hardware resources and web server implementation.
P.S. For paid clients, we offer Spaces replicas with round-robin load balancing. Essentially, your Space is duplicated, and a load balancer distributes the requests across the replicas.
1 Like
Hi @radames
thank you for the speedy response - this is super useful. Wow an amazing service. Out of interest, why do you call them demos as it appears you can scale to many thousands of users?
hi @harrycoppock for production inference we offer a more robust, autoscaling and dedicated infrastructure manage by us.