Six Ways an AI Phone Call Falls Apart

Out of a day of outbound calls to venues abroad, three of them were eventually handled by a human walking through the door in person. Not because the agent was badly built — I wrote about building it in I Built an AI Voice Agent That Makes Real Phone Calls, and the thing works — but because a phone line is a hostile environment and I had been quietly assuming it was a nice one.

The job was mundane. Call a list of venues in another country, ask about availability, write down the answers. In text, this is a solved problem. You send a message, it arrives or it doesn't, and if it doesn't you know within seconds. Voice has no such courtesy.

what actually happened — six failures in one day

Six distinct failure modes in a single afternoon. Not six instances of one bug. Six unrelated categories.

One: the line itself. A bad connection to a beach club produced constant dropouts. The agent asked its questions, half the syllables never arrived, and the person on the other end answered a question that hadn't been fully asked. No error was raised. The transcript looked like a conversation.

Two: crosstalk. At a restaurant, someone standing near the person who answered said something to a colleague. Speech recognition picked it up and attributed it to the callee. The agent then reasoned confidently on top of a sentence nobody had said to it.

Three: the one-turn hangup. A call connected, exchanged between one and three conversational turns, and died. Whether the human hung up, the carrier dropped it, or something in between failed, I still don't know. That's the point — from the agent's side, all three look identical.

Four: the phone menu. A venue answered with an automated menu in a language I don't speak. The platform documentation said it supported sending keypress tones. It accepted the instruction to send them. The tones never registered on the far end. The agent sat there politely listening to a menu it had no way to escape.

Five: hold queues. Put on hold, the agent waited, hit its timeout, and abandoned the call. Perfectly rational behaviour. Completely wrong outcome.

Six: the near-miss. On one attempt the agent gave up, and a human picked up exactly one ring later. One ring. That's the entire margin between a successful call and a wasted one.

Three separate venues were redialed. Each of the three failed again — and each failed in a new way, not the same way twice. That is the single most useful data point I took from the day. The failures weren't a bug I could chase down. They were a distribution.

the fix — and what wasn't fixable

The most expensive discovery was that outcome classification is itself unreliable. The agent repeatedly labelled a hold queue as voicemail, and hung up when the correct behaviour was to keep waiting. When your failure detector is one of your failure modes, no amount of retry logic saves you.

Then there was the capability list. Every one of these had to be granted explicitly, and none were on by default: the ability to hang up. The ability to detect voicemail. The ability to navigate a phone menu. The ability to switch languages mid-call. I found the first one during a live test that ended in dead air — the agent had finished its business and simply had no mechanism to end the call. It sat on an open line with a real human until I killed it manually.

The voice selection was worse, because it failed silently. I configured a specific voice. The conversational voice platform accepted the identifier at configuration time without validating it, then at call time fell back to a different voice — a different gender than intended. Nothing logged. Nothing flagged. It was caught only because a human being happened to be listening to that call live. Every text pipeline I've built would have thrown on that config. This one shrugged.

And one problem had no fix at all. The obvious remedy for poor international connection quality is "call from a local number." The telephony provider prohibits outbound calls from certain countries' geographic numbers outright, even routed through a trunk. Not a pricing tier. Not a feature request. A regulatory wall. The plausible fix was simply not legal to implement.

So the three stubborn venues were abandoned and handled in person on arrival. That was not a defeat. That was the plan working.

five questions to ask a voice-AI vendor

Everything above turns into a short evaluation script. I now run this before signing anything.

1. What is your success rate on outbound calls to numbers you don't control? Not inbound. Not a demo line. Cold outbound to a small business in another country. If they quote you a number above 80% without qualification, they are describing a demo.
2. Which capabilities are off by default, and what is the behaviour when one is missing? Hang up, voicemail detection, menu navigation, language switching. Ask them to name what happens when the agent lacks the ability to end a call. If they haven't thought about it, they haven't run enough live calls.
3. Do you validate configuration at config time or at call time? A voice identifier accepted silently and substituted at runtime is the tell. Ask what else fails open rather than loud.
4. How do you classify call outcomes, and what is your false-positive rate on voicemail detection? Hold queue versus voicemail is the hard case and it costs you real conversations. A vendor with no number here has never measured it.
5. Where are you legally prohibited from originating calls, and does that change if I bring my own trunk? Get this in writing, per country, before you design around a local number you can't have.

why this bites harder in the Gulf

Two things make this sharper regionally. First, almost nothing here happens in one language. A single afternoon of calls can cross Arabic, English, Hindi, and Tagalog — sometimes within one conversation, when the person answering switches to whoever is standing beside them. Language switching is not a nice-to-have feature you enable later. It is the default operating condition, and it interacts badly with crosstalk, because now your recognition layer is guessing at both who is speaking and which language they're in.

Second, numbering regulation across the region is genuinely restrictive and it varies country by country. The architecture that works from one jurisdiction is illegal from the one next door. If your voice vendor's answer to connection quality is "originate locally," you need to know — before you build — which of your markets that is actually available in. For a lean team, discovering this at deployment rather than at design is a rebuild, not a config change.

the rule that saved the money

One governing rule made the whole day cheap instead of expensive: every redial required a fresh brief and fresh explicit human approval. No automatic retry loops. Ever.

"Approved to call" never silently expanded into "approved to call again." Each redial was a new decision by a person who had read what happened on the previous attempt. That is why three failed venues cost me three extra calls and a walk-in, rather than forty calls and an irritated business owner who now associates my client with a robot that won't stop phoning.

This is the part people get wrong when they port text-automation habits to voice. Retrying a failed API request is free and invisible. Retrying a phone call spends money and interrupts a human being at work. Those are not the same operation and they do not deserve the same default.

Voice is not text with audio attached. Text automation fails in a handful of ways, most of them yours. Telephony fails in dozens, most of them environmental and entirely outside your control — the line, the room, the menu, the language, the person. Budget for a real-world success rate well below whatever the demo suggested. Then name a human fallback as the acknowledged plan B, up front, rather than iterating forever toward a number you're never going to hit. The subtler cousin of this problem — silence nobody notices until it's expensive — shows up in Five Seconds of Dead Air.

I now run explicit-approval-per-attempt and a named human fallback as standard in every voice engagement I take on. It's the first thing I write into scope, before any model or platform choice.

If you're weighing up a voice agent for outbound calls — booking, follow-ups, supplier chasing — I built and ran one through exactly this mess before I advise anyone else on it. For your business, in Dubai or across the UAE, a consulting day covers the five vendor questions above against whichever platform is in front of you, plus the approval rules and fallback plan written into scope. An hour of that usually saves a quarter of rebuild. Book a consulting day or send me an inquiry first if you'd rather talk before booking.