03 · Case study · Real-time AI
In voice systems, latency is product quality.
A conversational voice application: speech recognition, LLM inference, and speech synthesis, orchestrated asynchronously over WebSockets. Built to feel like talking to a person, monitored like infrastructure.
Context
Three slow operations in sequence — hear, think, speak — with a human waiting on the other end. Run them one after another and the conversation feels broken. The product only works if the whole pipeline fits inside a latency budget a person finds natural.
Architecture
- User speaks0 ms
- speech recognition
- partial transcript
- LLM inference
- first tokens
- speech synthesis
- audio response
Decision
The pipeline runs asynchronously over WebSockets: recognition, inference, and synthesis overlap instead of waiting on each other. Each stage is monitored like infrastructure — in a real-time system, a slow stage is a product defect.
Constraint
No stage could be allowed to stall the conversation, and no latency figure could be assumed — the design had to overlap work and monitor every stage, without inventing targets that didn't exist.
Result
A voice system that holds a natural conversation, with latency treated as a feature of the product rather than a number on a dashboard.
Run them one after another and the conversation feels broken.
Lessons
- In voice, every unnecessary pause makes the product feel broken.
- Async overlap is a design decision, not an optimization.
- Monitor real-time stages like infrastructure, because they are.
Stack
Deepgram · Groq · ElevenLabs · Twilio · FastAPI