How to build an advanced
voice agent

1. Give the agent more context about your conversation

When your agent asks the user a question, you should send that question into agent_context. This is really helpful for the AI model, as it can use this context to figure out what the user is likely saying - especially when the audio is unclear or muffled.

For example, if the agent has just asked "what's your email?", there's the risk that the model could write it as "user at example dot com". But if you include the agent_context, the agent will write user@example.com instead. The same approach works for names, dates, account numbers, and confirmations - anything where you know the rough shape of what's coming.

If you've got specific names, products, or terms that come up often and the model is unlikely to know them, you can use keyterms_prompt to improve their recognition. The model will use this list of words when it hears something ambiguous and ensures that it makes fewer mistakes. For example, "Vercel" might come out as "Ver sell" without the keyterms_prompt.

2. Focus the agent on the right speaker

You can use voice_focus to filter out any background noise and lock onto the primary speaker.

You can also select two modes for the voice focus, which allow you to handle different audiences and environments:

3. Stop the agent cutting you off mid-sentence

You can use turn_detection to control when the agent thinks you've finished speaking. By default, it uses a combination of silence, tonality, and pacing to work this out - which works well for most cases. But if the agent feels a bit too jumpy and is cutting you off, or is too slow and waiting awkwardly, you can adjust the settings.

To do this, you can change:

For example, if you're building a coaching agent or anything where the user pauses to think, you should set min_turn_silence a bit higher (around 800 to 1000ms) so that the agent waits before responding. But if you're developing a voice agent for drive-throughs or somewhere people speak quickly, you can keep this low.

4. Shape the agent's personality and give clear instructions

You can use system_prompt to give the agent a clear idea of what its role is, how it should respond to users, and what it should do. It's another important aspect, as it shapes the personality and behaviour of your agent.

For my coach build, the system prompt does most of the work for me. It tells the agent to act as an interview coach, keep responses to two sentences, push back on weak answers, and give structured feedback at the end. Without those instructions, the LLM defaults to giving long-winded essays that aren't well suited for voice.

When you're writing your own prompt, here are a few things you should consider adding:

• Always cap the response length explicitly - e.g. "respond with a maximum of two sentences"
• Tell it to ask one question at a time, rather than stack them up and overwhelm the user
• Tell it to push back on ideas when necessary, rather than simply agreeing by default
• Be specific about the format it should use

Here are a few example system prompts you can adapt as a starting point:

If you want to step this up a notch, you can also change the system prompt mid-session - which is really useful for agents that have to handle a wide range of conversations.

For example, a customer support agent could start with a general triage prompt that handles common questions, then swap to a more specialised prompt when the conversation moves into billing or technical topics. The conversation history stays intact, so that the user doesn't have to constantly repeat themselves.

5. Give the agent memory

For the agent to hold a proper conversation, it needs to remember what's already been said. The way this works is that you keep a running list of the conversation and send it to Claude on every turn, so each reply has the full context behind it.

This works really well for short sessions, like preparing for interview questions or brainstorming a single decision. For longer sessions that span several days, you'll want to summarise the older parts of the conversation yourself and feed that summary back in via the system prompt at the start of each session, so the history doesn't grow forever.

6. Include the expected language

While this isn't necessary and the agent can automatically detect the language for you, it is good practice and helps to eliminate any unnecessary mistakes.

To pin one, set language_code to the two-letter code for your target language:

It's also worth mentioning SpeakerRevision, which allows you to label different speakers in the same conversation, with the agent correcting any earlier mistakes as it learns more about each voice. This is really useful if you're building something like a meeting notetaker, where you need to know who said what and manage several different speakers.

1. Don't overlook the personal use cases

While a lot of people focus on business use cases - of which there are plenty - don't forget to explore how this could help you personally and change the way you work. We did this by building an AI voice coach that helps us prepare for upcoming interviews, make important decisions and explore them in greater detail, or brainstorm different ideas.

I've become quite tired of the chatbot experience in recent months, which is why I was keen to build my own thinking partner and talk through the ideas instead.

It'll be interesting to see how this evolves in the coming years, but I do think this is an emerging area that will only become bigger as LLMs do more tasks for us.

2. You don't need to be technical to build your own voice agent

Technology has changed a lot recently and you no longer need to write the code yourself, especially if it's a pretty straightforward idea. Claude Code is a great tool, as you can simply paste AssemblyAI's documentation and ask it to build a voice agent for you.

It's incredibly easy to get started and the technical aspect is not a huge barrier anymore for testing ideas. Instead, it's deciding what you want it to do and how you can scale that efficiently.

3. The system prompt does most of the work

I showed you several important settings and how you can tweak them to get better results from your voice agent. They're worth bearing in mind, as they give you extra control on top of the prompt.

But the core prompt is important for shaping your agent and how it interacts with people. As you start to use the agent more often and see how it performs, you can iterate on this prompt over time and constantly improve it.

4. Don't over-engineer it

Voice agents have a reputation for being complicated, but most of that comes from teams that got stuck in framework conversations before they'd even built anything.

Instead, you should start small and test your ideas. As I've shown already, you can do that in just 30 minutes and build on the idea from there.

Once you've got the prototype working well and consistently, you can start thinking about adding more complexity.