A voice agent answers every turn it hears. In a one-to-one call that is exactly right: you speak, it replies. Put the same agent in a group meeting and it becomes a problem. Every sentence anyone says is a turn, so the agent answers questions meant for other people and talks over the room.
I saw this up close recently, and the fix turned out to be simple enough to share. This post covers why the obvious fix does not work, what does, a template you can copy, and a five-step test to run before you bring the agent to a real meeting.
Why “only speak when your name is called” does not hold
The first thing everyone tries is a line in the prompt: only speak when someone says your name. I have tried it too. It lowers how often the agent interrupts. It does not stop it.
The reason is how voice agents are built. The whole system is tuned to notice that someone has finished a sentence and to produce a reply. A prompt rule asks the model to hear a turn, start to respond, and then choose silence. Most of the time it does. Some of the time it does not, and in a thirty-minute meeting with four people talking, “some of the time” comes up often.
Here is the real case. A team building a support assistant sent an avatar agent into their team Google Meet as a new team member. The idea was good: let it sit in on the meeting and answer when asked. In practice it was told to be quiet four times in three minutes, and it kept starting to answer. Nobody could finish a thought.
We rebuilt it the same day so that it stays silent until called. The prompt barely changed. The structure did.
The fix: two modes, and one of them cannot make a sound
Give the agent two sides. One is a speaking side that answers when called. The other is a silent observer that hears the meeting but cannot make a sound, because the platform mutes it. This is the part that matters. The observer is not asked to stay quiet. It has no way to speak.
The agent sits in the observer for most of the meeting. When someone says its name with a question, the observer hands over to the speaking side. The speaking side answers, and then it goes back to silent.
On Tough Tongue AI, the feature behind this is a Super Agent scenario with one sub-agent whose model is set to Standby. Standby is the silent observer. You do not need to remember the names to use it, because the template below is already wired this way.
What it does in the meeting
The template agent is called Maya, so I will use that name. Here is how she behaves:
- She joins silently. No greeting, no introduction.
- Say “Maya” plus a question and she answers in a sentence or three, then goes quiet.
- Say “Maya, be quiet” while she is speaking and she stops mid-sentence, without replying.
- Mention her name without asking anything (“Maya joined us today”) and she stays silent.
- She stays until the host removes her. You can also set her to leave when asked.
The short answers are deliberate. A participant who is called on in a meeting and then talks for two minutes is not much better than one who interrupts. A sentence or three, then silence, is how a good colleague answers when you turn to them mid-discussion.
How to get one
There are two ways to build it, and then one step to send it into a meeting.
Copy the template. There is a public template called Meeting participant (silent until called). Open it while signed in, open the three-dot menu, and choose Remix. Then replace two things: the name, and the FACTS section. The name is what people will say in the meeting, so pick one that is easy to pronounce and unlikely to come up by accident. The FACTS section is what your agent knows: your product, your pricing, your process, whatever it should be the expert on.
Or ask your coding agent. If you have the Tough Tongue AI MCP server and skills installed in Claude, Codex, or Cursor, say something like:
Build an agent that sits in our Meet and only talks when we say its name.The skill knows this pattern now, so it builds the two-mode structure instead of a prompt rule.
Send it into the meeting. Open the scenario’s meeting page and paste the Google Meet or Zoom link. If you want it to join on its own, connect a calendar instead.
Test it before a real meeting
Do not find out in front of a customer. Get one colleague, start a call, send the agent in, and run these five steps in order.
- 1Talk to each otherThe agent stays silent.
- 2Say its name plus a questionIt answers, then goes quiet.
- 3Talk to each other againSilent again.
- 4Say its name plus “be quiet” while it speaksIt stops mid-sentence.
- 5Say its name with no questionIt stays silent.
Steps one and three are the ones the prompt-only version fails. Step five is the subtle one. People say a teammate’s name all the time without wanting an answer, and the agent has to tell the difference.
Honest limits
This works well, and it is not perfect. Four things to know before you rely on it.
Deciding whether a sentence was meant for the agent is a model judgment. In a noisy room, or when the name is misheard, expect an occasional missed wake or false wake. A distinct name helps.
There is no chime or visual cue when it wakes yet. You find out it heard you when it starts to answer.
Each new question needs the name. You cannot call it once and then have a back-and-forth. That is the price of it not jumping in, and I think it is the right default for a group, but it feels a little formal at first.
The agent uses minutes for the whole time it is in the meeting, silent or not. It is listening the entire time, and listening is the work.
How other tools handle it
We are not the only ones who ran into this, and the answers look alike.
Tavus has a Wake Phrase setting: the agent stays silent until it hears a phrase you choose (see their changelog). Fireflies stays muted until it hears “Hey Fireflies”, then plays a chime. Otter uses “Hey Otter”. And if you are building your own pipeline, Pipecat ships a wake-phrase turn strategy.
The shared idea is the same everywhere: silent by default, a name to wake it. Our version differs in two ways. The wake name is the agent’s own name, in any language, so you say “Maya” the way you would say a colleague’s name. And the agent is a full voice agent that you define, with your facts, your voice, and your avatar.
Who this is for
Three uses have come up so far.
The first is a product or subject expert on call in sales and customer meetings. Your rep runs the call. When a question comes up about a limit, an integration, or a policy, they ask the agent by name and get the answer in the room instead of in a follow-up email.
The second is an AI new hire in team stand-ups. This was the original case. The agent sits in, hears everything, and answers when the team turns to it.
The third is a coach or interpreter that waits to be asked. Some help is only welcome on request.
If you already have a voice agent that works well one-to-one, this is the missing piece for bringing it into a room with more than one person. Copy the template, change the name and the facts, and run the five-step test with a colleague. If you are new to meeting agents, my guide on how to build an AI meeting agent covers how they join calls and what you get afterward, and you can start on Tough Tongue AI.
