Handling streaming connections at scale

Long-lived SSE connections need attention:

• Async, non-blocking I/O. Use a server model that handles many concurrent open connections cheaply (event-loop or async runtimes), since most connections are just waiting on the model. A thread-per-connection model would exhaust threads quickly.
• Proxy configuration. Reverse proxies and load balancers must allow long-lived streaming connections and not buffer them (or the real-time feel dies). Configure timeouts and buffering accordingly.
• Connection cleanup. When a client disconnects mid-stream, release resources and abort the in-flight model call so abandoned turns stop consuming tokens and compute.
• Graceful shutdown / deploys. Draining a server with open streams needs care so in-flight turns finish or fail cleanly rather than being cut off.