9 Ways an AI Voice Call Breaks in Production — And What We Did About Each One

By Ashish Aman · 25 August 2026 · 11 min read

Most people evaluating AI voice agents have already heard a bad one. The pause that runs a beat too long. The interruption that lands on top of you. The voice that sounds like it is reading. Vendors in this category do not discuss any of it. This is an account of nine specific ways Marklinea's voice agent has broken in production, what caused each one, and what we did — including the ones still open.

1. The visitor clicks, and hears nothing at all

A visitor gives their email, clicks to talk, and the panel says “Connecting you to Chloe — this will only take a moment.”

Then it says that forever.

No error, no timeout, no fallback. The single worst state a product can be in, because the visitor cannot tell whether to wait five more seconds or leave. Most leave, and they leave with a specific impression: this does not work.

The cause is a gap between two systems that both think they succeeded. Creating a call has two steps — a room gets created, then a worker has to pick up the job and join it. The first step returns success immediately. If no worker takes the job, nothing else happens. There is no error to report because nothing errored. The room is sitting there, correctly, empty.

What we did. The diagnostic that matters is checking whether the call-creation request returned at all, and what it returned. Three different faults present identically as a hang: a validation rejection because a required field was missing, a successful room creation with no worker to join it, and a frontend that never fired the request in the first place. They look the same to the visitor and they need completely different fixes.

We also took the button off the site rather than leave it hanging. A button that hangs with no way out is worse than no button, and shipping the broken version while we diagnosed would have cost more than the two weeks of not having it.

What we would do differently: put a client-side timeout on the connect sequence from the first day. Ten seconds, then an honest message and a booking link. The engineering fix took a while. The honest failure message would have taken an hour and would have saved every visitor in between.

2. One call is one worker, and workers are not free

This is the constraint nobody selling voice AI mentions, and it shapes everything.

A live voice call is not a web request. It is a persistent session holding a speech-to-text stream, a language model context and a text-to-speech connection open simultaneously, for as long as the human keeps talking. One call is one room is one worker process, holding gigabytes, for minutes.

Our workers also had a memory leak. Consumption climbed across calls rather than returning to baseline, until the process was killed and restarted. We raised the memory ceiling, which stopped the crashes and did not fix the leak. That distinction matters and we want to be straight about it: raising a limit buys time, it does not remove a fault. The underlying accumulation between calls is still unfixed.

What we did. Raised the ceiling as an immediate mitigation. Then designed the real fix, which is architectural rather than a patch: separate the cheap part of Chloe from the expensive part. A visitor arriving on the page gets an ambient greeting and a text conversation that costs a fraction of a cent. Only a visitor who has given an email and asked to talk escalates into a voice session. A pageview must never be able to spin up a voice worker.

Still open: the leak itself. Marked honestly rather than quietly.

3. She stops mid-sentence because a door closed

Barge-in is the feature that makes a voice agent feel like a conversation instead of a recording. When you start talking, she stops. Without it you get the hold-music experience of talking over a machine that will not yield.

With it tuned too sensitively, you get ours: Chloe would stop mid-sentence because of a small background noise. A door, a cough, someone else in the room, a notification. The prospect had not spoken. She stopped anyway, waited for input that never came, and the conversation stalled in a way neither party could explain.

This is the hardest problem on this list, and it is a genuine tradeoff rather than a bug with a correct answer. Voice-activity detection that is too permissive makes her talk over people. Too aggressive and a chair scrape derails a sentence. The right setting is different in a quiet home office and an open-plan sales floor, and you do not know which one your prospect is in.

What we did. Not enough yet, and the reason is worth stating. We deferred the tuning until we had enough real call data to know whether it was still happening at the rate we thought — because tuning a threshold against a hunch produces a different wrong threshold. Still open.

4. She decided the call was over

Chloe has a tool that ends the call, because someone has to close the conversation and a call that never ends is its own failure.

She used it too readily. A prospect would finish answering, start to ask a follow-up, and find the call already wrapping. From the prospect's side this reads as being hung up on — which is a memorable way to end a first conversation, in the wrong direction.

The cause is that “the conversation is complete” is a judgement, not a state. The agent had collected what it was asked to collect, and inferred that meant done. The prospect's remaining questions were not part of the completion criteria because nobody had made them part of it.

What we did. Made the ending condition explicit rather than inferred, so finishing the script is not the same as finishing the conversation. Verified by taking the call and deliberately asking a follow-up after the natural end.

5. She read the greeting at 201 words per minute

Our first recorded greeting sounded rushed. That was the whole bug report, and it is a useless one — “sounds rushed” is arguable, and two people will disagree about it forever.

So we measured. Two lines of greeting, read by the speech engine at 180 and 201 words per minute, against a conversational range of roughly 140 to 160. Not a matter of taste. A number, well outside the range, twice.

The fix is less obvious than it looks. The voice model has no speed control, so the audio has to be slowed after generation. But one global adjustment could not fix both lines, because they were rushed by different amounts — one needed a 12% slowdown and the other 20%. Applying an average would have left the first slightly slow and the second still fast.

What we did. Per-line tempo adjustment, and the generator now prints the words-per-minute of every take so “too fast” stays a number rather than an opinion. We also hash the approved audio, because a text-to-speech generator produces a slightly different take every run — without a digest, “the take you approved” and “the take that shipped” are two different objects that nobody can tell apart.

The transferable lesson: prefer measurements to adjectives. Nearly every voice quality complaint decomposes into a number once you look for one.

6. Nobody can say an email address out loud

Chloe needs the visitor's work email, because the entire product downstream — the transcript, the scorecard, the handover — is delivered to it. An email that arrives wrong does not fail loudly. It fails silently, and the prospect simply never receives the thing you promised them.

Speech-to-text is very good and it is not good at this. “ashish at marklinea dot com” has half a dozen plausible transcriptions. Names are worse. A single wrong character routes the entire deliverable into nothing.

What we did. Made it a rule with no exceptions: email is typed, never spoken. Chloe can ask by voice — asking is conversation — but the answer goes into a field that renders on screen. She never parses an address out of a transcript.

This forced a real architectural consequence. The stream between the agent and the browser could no longer be text alone; it had to carry structured events, so the agent can say render an email field now and the browser can do it mid-sentence. That is a meaningful amount of plumbing for one field, and it is the correct trade. The address is the product's delivery mechanism. It is the last thing to leave to a transcription.

7. The right greeting, the wrong script

[This section describes a fault involving another tenant on the same deployment. Names and script contents are deliberately omitted.]

This is the one we are least proud of, and the most instructive.

For three days after launch, every visitor who took the Chloe demo on our own website was greeted correctly — “Chloe from Marklinea” — and then asked a series of questions from an entirely different customer's discovery script. Wrong industry, wrong vocabulary, wrong problems.

The correct greeting is exactly why it took three days to notice. Identity resolved perfectly. Everything a casual check would look at looked right.

The cause was not a stale prompt. It was that the production deployment had no per-tenant routing at all — the script number was hardcoded, and there was no slot for ours to point at. Which means the obvious remedy, updating the prompt, would have changed nothing and cost an evening. The selector was broken, not the content.

What we did. Fixed the selection rather than the prompt, and shipped only that — resisting a full merge that would have carried another customer's unreviewed changes into their live agent as a side effect of fixing ours.

And then fixed the class, which is the part worth copying. Nothing in our system could have caught this. No endpoint reporting which configuration was actually loaded. No startup line naming the resolved tenant and script. No post-deploy check asserting that our tenant maps to our script.

Any one of those turns “a customer finds it in three days” into “the deploy fails in thirty seconds.” If you run a multi-tenant agent, make it say out loud who it thinks it is every time it starts. We did not, and we paid three days for it.

8. The fallback that quietly became someone else

Every serious voice stack has a fallback voice provider, because speech synthesis occasionally stumbles and finishing the sentence in a different voice beats not finishing it.

Ours had a default voice ID for tenants that had not specified one. That default was a real voice belonging to another tenant.

So if the primary provider stumbled mid-call, the fallback engaged — silently, automatically, exactly as designed — and Chloe would finish her sentence in a different company's persona. No error. No log line anyone would notice. Just a prospect on our call hearing someone else's agent take over mid-sentence.

The missing configuration field is not the defect. The default is. A tenant omitting a voice ID is an ordinary gap. Resolving that gap by borrowing another tenant's voice turns an ordinary gap into a silent cross-tenant leak.

What we did. Set the value explicitly, and — more importantly — changed what a missing value means. A missing voice ID now has to fail loudly at startup rather than fall through to a constant. A shared default must never point at any real tenant's anything.

The general rule we took from it: a fallback should degrade to something obviously generic, or refuse. It must never quietly resolve to something plausible and wrong. Plausible and wrong is the state you do not detect.

9. We could not tell whether the call had connected

A smaller one, and a good illustration of how a browser can refuse to answer a reasonable question.

When Chloe's call window closes, we want to say the right thing. “How did that go?” if it worked. Something more useful if it did not.

We could not find out which. The call runs in an embedded frame from a different origin, and a frame that is blocked outright still reports that it loaded. The one signal available says success in both cases. We wrote a watchdog to infer it from timing and removed it, because it was guessing and sometimes guessed wrong — which is worse than not asking, since a confidently incorrect “how did that go?” after a failed call is its own insult.

What we did. Wrote one message that is true in both cases: “How did that go? If it did not connect, or you want a longer session, book thirty minutes.” It acknowledges both outcomes without claiming to know which happened.

Sometimes the fix for a signal you cannot get is copy that does not need it.

What we would tell anyone evaluating a voice agent

Five questions. We would answer all five about ourselves, and the answers are above.

1. What happens when it fails to connect? Not what happens when it works. Ask to see the failure state. If the demo cannot show you one, they have not built one.
2. How many concurrent calls can it actually hold, and what happens at the ceiling? There is a ceiling. There is always a ceiling. The right answer is a graceful degrade to a booked human slot, not a hang.
3. How does it capture an email address? If the answer involves transcribing one, the delivery is unreliable and the vendor has not thought about it.
4. What does it do when someone coughs? Barge-in tuning is where a demo and a real sales floor diverge most.
5. If it is multi-tenant, how does it prove which configuration is loaded? If the answer is not an endpoint or a log line, they will eventually ship your script to someone else.

Why we published this

Three of the nine above are still open. We have said which.

The straightforward argument for not writing this is that a prospect reads it and worries. The argument for writing it is that every buyer evaluating voice AI has already heard a bad AI call, is already worried, and is currently choosing between vendors who all claim the problem does not exist.

We would rather be the one that names the failure modes and says what it did about each. Anyone can list features. The list above is harder to fake, and it is the more useful document.