A user opens a coaching session and says one word: “Hello.”
The agent replies: “Hi Ajitesh. Great to have you back. Today, we’ll dive into another classroom scenario and sharpen those escalation strategies together.”
Take that greeting apart, because more happened in it than in most product demos. The agent recognized a returning user by name (memory across sessions). It knew where the last session ended (state that persisted after the call dropped). Seconds later, it painted a picture of a seventh-grade classroom: an image generation tool call, live, mid-conversation. Then it presented a multiple-choice question about how to handle a disruptive student, a second tool, rendering structured UI inside a voice conversation. And when the user interrupted with “Could you explain the situation a bit more to me?”, the agent stopped speaking, rerouted, and elaborated without losing the thread.
None of this is a chatbot. A chatbot with a microphone bolted on is still a chatbot. This is a multimodal agent: audio in, audio out, images pulled in on demand, structured interactions rendered live, and a memory of who you are. This playbook is about building one.
What “multimodal” actually means
A multimodal agent perceives and produces across modalities in a single conversation loop.
On the input side: speech, first and foremost, with interruptions, accents, background noise, and false starts. Text as a fallback. Images and screen shares when the user wants to show the agent something. Uploaded documents. And structured events: a button click, an answer selected on a quiz card.
On the output side: synthesized speech with tone, pacing, and personality. Generated images, invoked as tools mid-conversation. Structured UI: option cards, quizzes, slides, scoring rubrics. And text artifacts: summaries, analyses, follow-up notes that outlive the call.
The critical point is that these are not separate features. In a good agent they interleave in one conversation. The agent speaks, draws, quizzes, listens, and remembers, and the user experiences it as one continuous interaction, the way a great teacher moves between talking, sketching on the whiteboard, and asking you a question without any of it feeling like a mode switch.
Voice is not text
In chat, the user can re-read, scroll up, copy a number. In voice, every word exists for one moment and then it is gone. That single fact reshapes almost everything about design.
Keep turns short: one idea or one question per turn. Front-load the important part of each sentence, because attention decays as the sentence goes on. Never assume the user remembers what was said three turns ago; they were listening, not reading. And respect the latency wall: above roughly one second of silence, presence breaks. Above two seconds, users start talking over the agent, and the conversation degrades from there.
Text agents optimize for correctness. Voice agents optimize for presence: latency, turn-taking, and tone matter as much as the words. A factually perfect answer delivered three seconds late, in a flat voice, loses to a good answer delivered instantly with the right energy.
But multimodal agents get an escape hatch that chat never had: when words are the wrong medium, switch modality. Describing a classroom seating chart in speech is painful. Generating an image of it takes one tool call. Listing four options aloud overloads working memory. The user has forgotten option one by the time you finish option four. Rendering them as a card on screen does not.
A rule of thumb you will see enforced throughout this playbook: if it takes more than two sentences to say, consider showing it instead.
The conversation loop
Here is what happens in one full turn of a voice conversation.
The user speaks, and speech-to-text (STT) transcribes incrementally, not waiting for the sentence to finish. The transcript, the conversation state, and the system prompt go to the language model, which reasons about what to do. The model may emit a tool call (generate an image, search memory, show a quiz) and the result feeds back into its context. The response text streams to text-to-speech (TTS), which often begins speaking before the full response is even generated. And the entire time, the pipeline listens for barge-in, the user interrupting, and must cancel the speech and roll back gracefully when it happens.
All of that has to complete in under a second to feel natural.
There are two broad ways to build this loop, and Chapter 2 covers them in detail. The cascade architecture chains separate STT, LLM, and TTS components: maximum control, more integration work. The realtime or speech-native architecture uses a single model that consumes and produces audio directly, with lower latency and more natural prosody, but less control over each stage.
Either way, the loop imposes a discipline that will recur in every chapter: every capability you add trades against latency. Vision, image generation, knowledge retrieval, tool calls: each one costs milliseconds inside a budget of about a thousand. Multimodal design is a budget exercise, and the builders who internalize that early ship agents that feel alive.
Where the field is, and where this playbook fits
A short timeline explains why this playbook is organized the way it is.
In 2023, AI entered the mainstream and the craft was prompt engineering. What exactly do you say to the model? The context was the prompt.
In 2024, tools arrived. Agents could act (book, search, generate) and the craft became the interaction between the prompt and the tool instructions.
In 2025, builders started assembling whole agents: system prompts plus referential memory plus knowledge bases plus external tool servers plus sub-agents. The craft became context engineering, designing everything around the model, not just the words sent to it.
In 2026, multimodal is the default. Voice, vision, and image generation are table stakes on every serious platform. The differentiator is how well the whole workspace is engineered, which is exactly what the rest of this playbook covers: the stack (Part 1), the workspace (Part 2), memory (Part 3), sub-agents (Part 4), conversation design (Part 5), iteration (Part 6), and production (Part 7).
The examples throughout come from building on Tough Tongue AI, because that is where we build daily, but every pattern is platform-agnostic, and we will flag the places where platforms differ.
Before engineering the workspace, though, you need to understand the machinery underneath. Next: the multimodal stack.