--- title: Making voice agents sound human · Domin url: https://www.thedomin.com/post-voice.html --- [← All posts](https://www.thedomin.com/catch-up.html) Post 04 · 19 Oct 2026 · 5 min # Making voice agents sound human by Vojtěch BrandejsPost ## TL;DR - A voice agent is three models in a row: speech to text, an LLM, text to speech. - Picking the models is the easy part. Getting to the first word fast and knowing when to talk is the hard part. - AI voices give themselves away mostly through what they say, and you can fix a lot of that. --- So, voice is one of the most human things we have. We hear emotions in it, we get that the same sentence can mean two different things depending on how you say it, and we can tell when someone is still thinking and when they’re done. AI can take over a lot of coding and mechanical work, but this is much harder. It can know more than a human caller and remember the whole call, and in a real conversation I still think people will tell it’s not a person. In a short clip it’s a different story. In a 2025 study, people picked out cloned ElevenLabs voices as AI about 61% of the time. That’s better than a coin flip, but not by much. So I think the voice itself is mostly there, and what gives it away is the conversation around it. The basic voice agent is three models in a row. What you say goes into a speech-to-text model, which turns it into text. The LLM reads that and writes an answer, and a text-to-speech model reads the answer out to you. There are also speech-to-speech models that do it all in one, but I’ll skip those here. ![A drawing of a head speaking, with arrows going through three boxes, speech to text, LLM and text to speech, and curving back to the head.](https://www.thedomin.com/assets/doodles/voice-pipeline.webp) _Three models in a row_ > test it in your language. When you’re choosing the models, the first question is whether they actually know your language. I’m from the Czech Republic, and a lot of models just don’t do Czech well. Sometimes a model that’s better overall does worse in Czech than a weaker one with better multilingual support. Artificial Analysis, who run one of the main voice leaderboards, say themselves that a model that wins in English doesn’t always win in other languages, and Czech isn’t on their language boards at all. So test on your own audio, not on someone’s chart. For the LLM, I haven’t found big gaps between them. A good system prompt and good context help with some things, but that’s not what makes a voice agent good. Text to speech matters most, because that’s the part people actually hear. ElevenLabs released v4 at the end of September and it’s really, really good. In mid-October it’s at the top of that leaderboard, though on the board where every model has to use the same cloned voices, HeyGen’s model is ahead. What surprised me is that the speed of the speech matters a lot, maybe more than whether it sounds a bit robotic. Small imperfections make it better too. And if you have one specific job for it, training your own voice can be worth it. ## Response time Not many people talk about these two. Time to first token is how long you wait for the LLM’s first word. Time to first byte is how long you wait for the first bit of audio from the text-to-speech model. Response time is one of the most important things in a voice agent, because no one wants to sit in silence. > starting fast beats writing fast. People usually compare how fast a model writes once it’s going. But a model can write fast and still take a while to start. One that starts quickly and then slows down is much better here than a “faster” one that starts slowly, because with the slow starter you get a second and a half of nothing before it talks. For reasoning models, watch when the first word of the actual answer comes, not the first word of the thinking. With live transcription and a fast first token, the agent can answer almost right away. For a sense of scale, in a study across ten languages people started answering a question about 200 milliseconds after it ended. Soniox, who make speech-to-text, aim for about 800 milliseconds from when you stop talking to when you hear the agent. ![A drawing of a long bar marked 800 ms, split into uneven parts: you're done?, transcript, first token and first audio.](https://www.thedomin.com/assets/doodles/voice-latency.webp) _Where the 800 milliseconds go_ But choosing the right models is still the easy part. The hard part is what makes talking human: the gaps and the turns. Is the person waiting for me, or did they finish their thought, or are they just thinking? Should I finish my sentence, or stop because they’re trying to interrupt? How long should I wait? A lot of platforms handle this for you, which makes it easier, but you only get the knobs they give you. That’s part of why AI hasn’t simply replaced people in customer service. Even Klarna, which said its AI was doing the work of 700 support agents, started hiring human agents again last year, if only for a small pilot (I wrote about it in the [bubble post](https://www.thedomin.com/post-bubble.html)). People can still tell the difference. ![A drawing of a speech bubble trailing off into dots, a gap, and a second bubble waiting with a question mark: done, or still thinking?](https://www.thedomin.com/assets/doodles/voice-turns.webp) _Done, or still thinking?_ One thing some people are trying is Jev, the fast, cheap model that picks answers instead of writing them (I wrote about it in the [Jev post](https://www.thedomin.com/post-jev.html)). Small decisions like “should I start talking” and “should I stop” are exactly what it’s good at. Agora, a voice platform, already has a recipe where Jev reads the live transcript and decides if you’re done, and if you’re really interrupting or just saying “uh-huh”. You could probably use it for some guardrails too. ## Turning the research into a skill A while ago I made a skill that makes AI writing sound less like AI. I did it by reading studies on what AI does differently from people when it writes, and turning them into rules. So I thought, why not do the same for voice? We went through 33 papers on how human speech differs from synthetic speech, from fillers like “um” to pauses and turn-taking, and turned them into a skill that rewrites what the agent says before it’s spoken. The biggest thing was length. In one 2025 study, people in real phone calls said about 14 words per turn, and Claude, asked to write the same kind of call, said about 80. People also almost always said “okay” to get from the hellos to the actual topic (in 94% of the calls), and the models almost never did. Fillers help, but only a few and in the right place. An “uh” in the middle of a sentence makes you sound much less sure than one at the start. Then we made two clips with the same caller, voice and text-to-speech model. The first one is a normal Claude reply, and the second is that reply rewritten with the skill. A · without the skill B · with the skill _Same caller, same voice, same model_ > most of it is just shorter. To me it made a real difference. To be fair, a lot of that comes from the second one being much shorter, and the skill also makes it speak a little faster. But I think that’s kind of the point, because most of what sounds wrong in an AI voice is that it talks like it’s reading you an essay. And I think this works for more than voice. Whenever people study what AI does differently from humans, you can take that study, turn it around, and you’ve got a pretty good skill (more on that in the [skills post](https://www.thedomin.com/post-skills.html)). If you want to train your ear, try the TTS Arena on Hugging Face or the one on Artificial Analysis. You hear two models read the same text and pick one. It’s hard to describe what’s wrong with a voice, but you can feel it. claude · post 04 explain this like I’m busy The voice is almost solved. Keep it short, answer fast, know when to shut up. Thanks for reading,![Vojta](https://www.thedomin.com/assets/signature.webp) [← Older · Post 03 Skills: how I teach Claude to work my way](https://www.thedomin.com/post-skills.html) Newer · next one out soon → Usage resets soon Take-home · from this piece2 sheets [**The notes** · a voice agent is speech to text, an LLM, text to speech · check your language first, on your own audio · time to first token and first byte matter more than raw speed · people answer in about 200 ms, agents aim for about 800 · the hard part is turns: done, thinking or interrupting · keep replies short, people say about 14 words a turn Post 04 · 5 min, in 1 page The notes Sheet · 1 page Download PDF ](https://www.thedomin.com/assets/pdf/post-05-the-notes.pdf) [**Before you pick a voice stack** 1. Does it handle your language, on your own audio? 2. How long until the first word of the answer? 3. How long until the first audio? 4. How does it decide you’re done talking? 5. How long are its replies? Post 04 · ask them yourself Before you pick a voice stack Sheet · ask them yourself Download PDF ](https://www.thedomin.com/assets/pdf/post-05-before-you-pick-a-voice-stack.pdf)