Designing voice agents for latency people can live with

In a chat interface, a second of delay is fine. On a phone call, it is a lifetime. A voice agent that reasons brilliantly but responds slowly loses the caller before it finishes its first sentence. Latency is the constraint that shapes every other decision in a voice system.

This article walks through where the time goes in a voice turn, the architecture choices that win it back, and how to measure latency the way callers experience it. It is part of a series on voice agents; the overall system is mapped in Voice AI agent architecture: from phone line to response.

What callers expect

Human conversation runs on tight timing. Research on turn-taking across languages finds that the typical gap between one speaker finishing and the next starting is a fraction of a second, commonly around 200 milliseconds. Callers do not measure this consciously, but they feel it. Once a reply takes noticeably longer than a person would, the agent starts to sound hesitant, callers repeat themselves or talk over it, and the conversation breaks down.

A voice agent will not match that on every turn, and it does not have to. The goal is a response time that feels like a person thinking, not a machine processing. Consistency matters as much as the average: an agent that is usually quick but sometimes stalls for several seconds feels worse than one that is steady.

Published latency targets

It helps to have concrete numbers to design against. Twilio publishes benchmark targets for the median latency of a voice agent at launch, for a cascaded pipeline with separate speech-to-text, language model, and text-to-speech stages. They make a sensible starting budget.

Two things stand out. Even a well-built pipeline lands at around a second, well above the gaps in human conversation, so the techniques below are about how fast the agent feels as much as raw speed. And the component targets add up to less than the end-to-end figure: the difference is the phone network, endpointing, and the hops between services, which is exactly the overhead teams tend to forget when they benchmark components on their own.

Where the seconds go

A voice turn passes through several stages, and each adds delay. Understanding the budget means knowing what each stage costs before you try to cut it.

• Endpointing: deciding the caller has finished speaking. Wait too long and every turn starts late; cut in too early and the agent interrupts.
• Speech recognition: turning the caller's audio into text, ideally streamed while they speak.
• Reasoning: the language model deciding what to say or do next, including any tool calls to look up or update records. Measured as time to first token (TTFT).
• Speech synthesis: turning the response back into audio. Measured as time to first byte (TTFB) of audio, the moment the caller can start hearing it.
• Network and telephony: moving audio between the caller, the telephony provider, and each service in the pipeline.

The stage teams optimise first is usually the model, because it is the most visible. In practice the biggest losses are often in the gaps between stages: waiting for silence before acting, waiting for a whole response before speaking it, and sending audio across regions between services.

Stream everything

The single biggest win is to stop waiting for whole stages to finish. Stream recognition so reasoning can start before the caller stops speaking. Stream synthesis so the caller hears the start of an answer while the rest is still being produced. A pipeline that streams feels responsive even when the total work is the same.

Get turn-taking right

End-of-turn detection is where many voice agents lose the most time. The simplest approach waits for a fixed period of silence, but any period long enough to avoid cutting people off mid-thought adds that delay to every single turn. Better approaches combine voice activity detection with signals from the transcript itself, such as whether the sentence sounds complete, so the agent can respond quickly to a finished answer and wait patiently through a pause.

Barge-in is the other half. Callers interrupt, especially when the agent is telling them something they already know. The pipeline should detect speech while the agent is talking, stop playback immediately, and treat what the caller said as the next turn. An agent that keeps talking over a caller feels slower than one that is technically less responsive.

Keep heavy work off the critical path

Not every turn needs the most capable model, and not every turn needs a tool call. Routine turns such as confirmations and short answers can run on smaller, faster models, with stronger models reserved for turns that need real reasoning. When a turn does need a slow lookup, the agent can say something natural while it works, such as confirming what it is about to check, rather than leaving dead air.

Place services close to each other and to the telephony provider. Audio that crosses regions between recognition, reasoning, and synthesis adds delay at every hop, and those hops repeat on every turn.

Choose the voice for the phone line

Text-to-speech choice matters more than it looks. Providers such as ElevenLabs and Fish Audio, both of which we have integrated, stream synthesis and start speaking quickly, but the voice and model you pick still change time to first audio and how clear the agent sounds over narrowband phone audio. We compare candidate voices on real calls rather than in a studio demo, and keep the provider behind an interface so a better option can be swapped in later.

Measure what callers hear

Component benchmarks are a starting point, not the answer. What matters is the time from the moment the caller stops speaking to the moment they hear the agent's first sound, measured on real calls over real phone lines. We log timestamps at each stage of every turn so we can see where the time went, and we track the slow tail (the 95th and 99th percentiles, not only the median), because the occasional long pause is what callers remember, and the slow tail is where tool calls, cold starts, and slow network paths show up. Testing this before launch is covered in How to evaluate a voice agent before it takes real calls.

Know when to hand off

Not every call should be resolved by the agent. Deciding quickly that a call needs a person, and handing off with context, is part of a good latency experience. A fast, clean escalation beats a slow attempt to handle something the agent was never going to resolve. We cover how to design that handoff in When a voice agent should hand off to a person.

Practical recommendations

1. 01Set a latency budget for the full turn and hold every stage to its share.
2. 02Stream recognition, reasoning, and synthesis rather than waiting for each stage to finish.
3. 03Use end-of-turn detection that considers what was said, not only how long the silence lasted.
4. 04Support barge-in, and stop speaking the moment the caller does.
5. 05Route routine turns to faster models and keep slow lookups off the critical path.
6. 06Keep services in the same region as each other and the telephony provider.
7. 07Measure caller-perceived latency on real calls, including the slow tail.

If you are planning a voice agent and want a second opinion on the architecture, this is where our voice AI work starts.