Who’s running open-source LLMs in enterprise production, and how?

Inference Endpoints on the Hub
1

Hi everyone,

I wasn’t sure which category this best fits in, so I’m posting here since it’s about production deployment.

I’m trying to understand how enterprise teams are deploying open-source LLMs in real production environments.

If you’re running models internally or on your own infrastructure, I’d love to hear about your setup:

  • How you’re serving the model

  • The hardware or cloud configuration you’ve found viable

  • Key challenges you’ve hit (throughput, latency, cost, monitoring, compliance)

  • And what finally made your setup feel production-ready

I’m especially interested in enterprise use cases that have actually gone live versus those that stayed in prototype.

Feel free to share deep technical details or architecture notes if you can.

Thanks for taking the time to share your experience.

1 Like