A voice agent looks simple from the outside: someone calls, something answers. Underneath, it is a real-time distributed system with half a dozen moving parts, each with its own failure modes, and all of them have to cooperate within the timing of a natural conversation. This article maps the components of a production voice agent, what each one does, and the decisions that matter at each layer.
Telephony: getting the call in
Calls arrive over the public phone network. A telephony provider gives you phone numbers and connects calls to your system, typically through SIP trunking or a programmable voice API that streams call audio over a WebSocket. Calls from a browser or app can use WebRTC instead, which offers better audio quality and lower latency than a phone line. We most often build this layer on Twilio: numbers, SIP trunking, media streams that carry call audio to the agent over a WebSocket, and transfers to staff.
The decisions here are about operations as much as technology: which numbers and regions you need, how calls are routed when the agent is busy or unavailable, how transfers to staff work, and whether calls are recorded, under which consent rules. Standard phone audio is narrowband, commonly sampled at 8 kHz, which is lower quality than the audio most speech models are trained on. The rest of the pipeline has to be tested on real phone audio, not studio recordings.
The media pipeline: streaming audio both ways
Between the phone line and the AI services sits a media layer that streams audio in real time. It receives the caller's audio in small frames, detects when they are speaking, sends audio to recognition, and plays synthesised speech back. It also handles end-of-turn detection and barge-in, where the caller interrupts and the agent has to stop talking immediately.
This layer is where most of the conversational feel is won or lost. Turn-taking decisions made here add or remove delay on every turn, as we cover in Designing voice agents for latency people can live with.
Speech recognition and synthesis
Speech-to-text converts the caller's audio into a transcript, ideally streamed so partial results arrive while the caller is still speaking. Accuracy on names, numbers, addresses, and domain vocabulary matters more than general accuracy, because those are exactly the details a booking or enquiry depends on. Many recognition services accept vocabulary hints, and the agent should confirm critical details back to the caller.
Text-to-speech turns the agent's response into audio. The voice should be clear on a phone line, consistent across the call, and quick to start. Streaming synthesis, which begins playing the first words while the rest are still being generated, is what keeps responses from feeling delayed.
We have shipped integrations with ElevenLabs and Fish Audio at this layer. Both stream audio and support custom and cloned voices, so the choice comes down to the voice itself, the languages needed, latency over the phone line, and cost. We keep the provider behind an interface and test candidates on real calls before committing.
Orchestration: where the agent thinks
The orchestration layer holds the conversation state and decides what happens next. A language model interprets the caller's request and either responds or calls a tool: checking availability, looking up an account, creating a booking, or updating a record. Tools are ordinary functions with narrow permissions, so the agent can only do what the workflow allows.
Spoken conversation needs different instructions from text chat. Responses should be short, avoid lists and formatting that make no sense aloud, and confirm actions explicitly. State has to survive the whole call, including interruptions and changes of topic, and be available to a person if the call is transferred.
We keep model calls behind a provider-agnostic layer, so different turns can use different models and traffic can move to another provider if one slows down or fails. The same approach keeps cost predictable, as described in How we kept LeadVector's inference costs predictable.
Integrations, guardrails, and observability
A voice agent is only useful if it can act on real systems: calendars, scheduling tools, CRMs, ticketing, and knowledge bases. Integrations need the same care as any backend work, with timeouts, retries, and clear behaviour when a system is slow or down. Where answers come from documents rather than structured records, the agent uses retrieval, with the grounding concerns described in Evaluating grounded answers in RAG systems.
Guardrails define what the agent must never do or say, which topics it should decline, and when it must hand the call to a person. Observability records every call: transcripts, tool calls, timings for each stage, outcomes, and handoffs. Without that record you cannot tell why a call went wrong, and you cannot improve the agent safely.
Cascaded pipeline or speech-to-speech?
Everything above describes a cascaded pipeline: separate speech-to-text, language model, and text-to-speech stages. The alternative is a speech-to-speech model, which takes the caller's audio in and produces spoken audio out in a single model, as the realtime voice APIs from the major model providers now do.
Speech-to-speech removes two hand-offs, so it can respond faster, and it hears tone and emphasis that a transcript loses. The trade-offs are control and choice: fewer voices, less ability to inspect or correct what happens between hearing and speaking, and guardrails and tool calls that are harder to enforce mid-turn. Cost and model choice are also tied to one provider.
A cascaded pipeline lets you choose the best component at each stage, such as a recognition model that handles phone audio well and a voice from ElevenLabs or Fish Audio, and it gives you a transcript to check before anything is said or done. For most business call flows, where correctness, tool use, and auditability matter more than the last few hundred milliseconds, we still recommend cascaded. Speech-to-speech is worth testing where the conversation itself is the product, and the balance will shift as those models mature.
One agent, several channels
The orchestration layer does not have to be tied to the phone. The same agent definition, with the same tools and guardrails, can serve web chat, WhatsApp, and email, with only the input and output layer changing per channel. Our Cedarline demonstration build is built this way: one agent with shared tools across voice and three text channels.
Build or buy?
Hosted voice-agent platforms such as Retell AI bundle telephony, speech, and orchestration, and they are the fastest way to get a use case live. We have built on Retell AI where speed to launch mattered, with our own tools, guardrails, and integrations wired in behind it. The trade-offs tend to appear later: limited control over latency and turn-taking, difficulty integrating deeply with your own systems, per-minute pricing that grows with volume, and dependence on one vendor's model choices. A custom pipeline takes more engineering but gives you control of each layer. A common path is to validate the use case on a platform, then bring the layers that matter most in-house once call volume and requirements are clear.
Architecture checklist
- 01Choose telephony that supports real-time audio streaming and warm transfer to staff.
- 02Decide between a cascaded pipeline and speech-to-speech per use case, not by default.
- 03Test speech recognition on real phone audio, with your own names and terms.
- 04Stream recognition and synthesis, and support barge-in.
- 05Give the model narrow tools and instructions written for speech.
- 06Keep model calls provider-agnostic, with fallbacks.
- 07Design escalation paths before the happy path.
- 08Log every stage of every call, with timings.
If you are planning a voice agent, our voice AI page sets out how we approach the build.
References
Mudassar Iqbal
Founder and Principal Engineer, Valtair
Mudassar Iqbal is the founder of Valtair, where he designs, builds, and operates its products and client systems. He writes about what he learns shipping AI to production.