In a chat interface, a second of delay is fine. On a phone call, it is a lifetime. A voice agent that reasons brilliantly but responds slowly loses the caller before it finishes its first sentence. Latency is the constraint that shapes every other decision in a voice system.
Where the seconds go
A voice turn passes through several stages, and each adds delay. Understanding the budget means knowing what each stage costs before you try to cut it.
- Speech recognition: turning the caller's audio into text.
- Reasoning: deciding what to say or do next.
- Speech synthesis: turning the response back into audio.
- Network and telephony: moving audio to and from the caller.
Stream everything
The single biggest win is to stop waiting for whole stages to finish. Stream recognition so reasoning can start before the caller stops speaking. Stream synthesis so the caller hears the start of an answer while the rest is still being produced. A pipeline that streams feels responsive even when the total work is the same.
Know when to hand off
Not every call should be resolved by the agent. Deciding quickly that a call needs a person, and handing off with context, is part of a good latency experience. A fast, clean escalation beats a slow attempt to handle something the agent was never going to resolve.
Practical recommendations
- 01Set a latency budget and hold every stage to its share.
- 02Stream recognition, reasoning, and synthesis rather than waiting for each.
- 03Reserve heavy reasoning for the turns that need it.
- 04Escalate quickly and with context when a call is out of scope.
- 05Measure real call latency, not just component benchmarks.
References
Mudassar Iqbal
Founder and Principal Engineer, Valtair
Mudassar Iqbal is the founder of Valtair, where he designs, builds, and operates its products and client systems. He writes about what he learns shipping AI to production.