Making voice agents sound human
TL;DR
- A voice agent is three models in a row: speech to text, an LLM, text to speech.
- Picking the models is the easy part. Getting to the first word fast and knowing when to talk is the hard part.
- AI voices give themselves away mostly through what they say, and you can fix a lot of that.
So, voice is one of the most human things we have. We hear emotions in it, we get that the same sentence can mean two different things depending on how you say it, and we can tell when someone is still thinking and when they’re done. AI can take over a lot of coding and mechanical work, but this is much harder. It can know more than a human caller and remember the whole call, and in a real conversation I still think people will tell it’s not a person.
In a short clip it’s a different story. In a 2025 study, people picked out cloned ElevenLabs voices as AI about 61% of the time. That’s better than a coin flip, but not by much. So I think the voice itself is mostly there, and what gives it away is the conversation around it.
The basic voice agent is three models in a row. What you say goes into a speech-to-text model, which turns it into text. The LLM reads that and writes an answer, and a text-to-speech model reads the answer out to you. There are also speech-to-speech models that do it all in one, but I’ll skip those here.
When you’re choosing the models, the first question is whether they actually know your language. I’m from the Czech Republic, and a lot of models just don’t do Czech well. Sometimes a model that’s better overall does worse in Czech than a weaker one with better multilingual support. Artificial Analysis, who run one of the main voice leaderboards, say themselves that a model that wins in English doesn’t always win in other languages, and Czech isn’t on their language boards at all. So test on your own audio, not on someone’s chart.
For the LLM, I haven’t found big gaps between them. A good system prompt and good context help with some things, but that’s not what makes a voice agent good.
Text to speech matters most, because that’s the part people actually hear. ElevenLabs released v4 at the end of September and it’s really, really good. In mid-October it’s at the top of that leaderboard, though on the board where every model has to use the same cloned voices, HeyGen’s model is ahead. What surprised me is that the speed of the speech matters a lot, maybe more than whether it sounds a bit robotic. Small imperfections make it better too. And if you have one specific job for it, training your own voice can be worth it.
Response time
Not many people talk about these two. Time to first token is how long you wait for the LLM’s first word. Time to first byte is how long you wait for the first bit of audio from the text-to-speech model. Response time is one of the most important things in a voice agent, because no one wants to sit in silence.
People usually compare how fast a model writes once it’s going. But a model can write fast and still take a while to start. One that starts quickly and then slows down is much better here than a “faster” one that starts slowly, because with the slow starter you get a second and a half of nothing before it talks. For reasoning models, watch when the first word of the actual answer comes, not the first word of the thinking. With live transcription and a fast first token, the agent can answer almost right away.
For a sense of scale, in a study across ten languages people started answering a question about 200 milliseconds after it ended. Soniox, who make speech-to-text, aim for about 800 milliseconds from when you stop talking to when you hear the agent.
But choosing the right models is still the easy part. The hard part is what makes talking human: the gaps and the turns. Is the person waiting for me, or did they finish their thought, or are they just thinking? Should I finish my sentence, or stop because they’re trying to interrupt? How long should I wait? A lot of platforms handle this for you, which makes it easier, but you only get the knobs they give you.
That’s part of why AI hasn’t simply replaced people in customer service. Even Klarna, which said its AI was doing the work of 700 support agents, started hiring human agents again last year, if only for a small pilot (I wrote about it in the bubble post). People can still tell the difference.
One thing some people are trying is Jev, the fast, cheap model that picks answers instead of writing them (I wrote about it in the Jev post). Small decisions like “should I start talking” and “should I stop” are exactly what it’s good at. Agora, a voice platform, already has a recipe where Jev reads the live transcript and decides if you’re done, and if you’re really interrupting or just saying “uh-huh”. You could probably use it for some guardrails too.
Turning the research into a skill
A while ago I made a skill that makes AI writing sound less like AI. I did it by reading studies on what AI does differently from people when it writes, and turning them into rules. So I thought, why not do the same for voice?
We went through 33 papers on how human speech differs from synthetic speech, from fillers like “um” to pauses and turn-taking, and turned them into a skill that rewrites what the agent says before it’s spoken.
The biggest thing was length. In one 2025 study, people in real phone calls said about 14 words per turn, and Claude, asked to write the same kind of call, said about 80. People also almost always said “okay” to get from the hellos to the actual topic (in 94% of the calls), and the models almost never did. Fillers help, but only a few and in the right place. An “uh” in the middle of a sentence makes you sound much less sure than one at the start.
Then we made two clips with the same caller, voice and text-to-speech model. The first one is a normal Claude reply, and the second is that reply rewritten with the skill.
To me it made a real difference. To be fair, a lot of that comes from the second one being much shorter, and the skill also makes it speak a little faster. But I think that’s kind of the point, because most of what sounds wrong in an AI voice is that it talks like it’s reading you an essay.
And I think this works for more than voice. Whenever people study what AI does differently from humans, you can take that study, turn it around, and you’ve got a pretty good skill (more on that in the skills post).
If you want to train your ear, try the TTS Arena on Hugging Face or the one on Artificial Analysis. You hear two models read the same text and pick one. It’s hard to describe what’s wrong with a voice, but you can feel it.
explain this like I’m busy
The voice is almost solved. Keep it short, answer fast, know when to shut up.
Thanks for reading,