The most important design decision in a voice agent is not what it can handle. It is what it should not. Every voice agent will meet calls it cannot or should not resolve: a complaint, an urgent problem, a complicated exception, a caller who simply wants a person. How the agent recognises those calls, and what happens next, decides whether callers trust it.
We design escalation paths before the happy path. This article covers when to hand off, how to do it without making the caller start again, and how to tell whether the balance is right.
Why handoff design comes first
An agent that tries to handle everything fails in the worst way: slowly, after the caller has explained their problem twice. An agent that escalates too readily saves no one any time. The right balance depends on the business, and it should be a deliberate decision rather than whatever the model happens to do.
Starting with escalation also clarifies scope. Listing the calls that must go to a person usually shows that the agent's real job is narrower, and more achievable, than the original brief.
Signals that a call needs a person
- The caller asks for a person. This should always be honoured, quickly, without the agent trying to talk them out of it.
- The request is outside the agent's scope: a topic it was not built for, or an action it has no tool to perform.
- Something is at stake: a safety issue, a legal or medical question, a complaint, a payment dispute, or a cancellation the business wants a person to handle.
- The conversation is not progressing: the agent has failed to understand the same point more than once, or the caller keeps repeating themselves.
- The caller is frustrated or distressed, shown by what they say and how the conversation is going.
- A system the agent depends on is unavailable, so it cannot complete a task it would normally handle.
Some of these are rules, written into the agent's instructions and enforced in code. Others are judgements the model makes during the call. Anything with real consequences should be covered by a rule rather than left to judgement alone.
Warm handoff: transfer the context, not just the call
A cold transfer, where the caller lands in a queue and starts from the beginning, throws away the conversation the agent just had. A warm handoff passes the context along: who is calling, what they want, what the agent has already checked or done, and why it is escalating.
In practice the agent writes a short structured summary to the CRM or support tool before transferring, and, where the telephony allows it, the staff member sees that summary as the call arrives. The agent should also tell the caller what is happening: that it is passing them to a colleague, and that they will not need to repeat everything.
On Twilio, for example, a warm transfer is typically built as a conference: the staff member joins with the summary in front of them, and the agent leaves the call once they have taken over. Hosted platforms such as Retell AI provide their own transfer options, and the same rule applies: the context must travel with the call.
When no one is available
Escalation paths have to work outside business hours and when every staff member is busy. The options are a scheduled callback, a message taken by the agent and logged with full context, or a clear explanation of when someone will be available. Each needs a real process behind it: a callback promised and never made is worse than no agent at all.
Measuring the balance
Escalation is something to measure, not set once. Useful measures include the share of calls handed off and the reasons, how many handed-off calls the agent could have resolved, how many resolved calls should have been escalated, and whether transferred callers had to repeat themselves. Reviewing transcripts from both groups regularly shows where to adjust the rules, the instructions, or the tools.
These checks belong in the evaluation set too, with simulated calls that should and should not escalate, as described in How to evaluate a voice agent before it takes real calls.
Practical recommendations
- 01List the calls that must go to a person before designing anything else.
- 02Always honour a request for a person, immediately.
- 03Enforce consequential escalation rules in code, not only in the prompt.
- 04Hand off with a structured summary so the caller never repeats themselves.
- 05Tell the caller what is happening and what comes next.
- 06Have a real out-of-hours path, with a process behind every callback.
- 07Measure both missed escalations and unnecessary ones, and review transcripts regularly.
Handoff design is part of how we approach voice AI, and of agents with human approval more broadly.
Mudassar Iqbal
Founder and Principal Engineer, Valtair
Mudassar Iqbal is the founder of Valtair, where he designs, builds, and operates its products and client systems. He writes about what he learns shipping AI to production.