Take a moment to appreciate what happened in Chapter 1’s demo. A builder wrote some English into a text box, and out came an agent with memory, state, image generation, and a personality. It feels like magic.
It is actually four layers of stack, each solving a distinct engineering problem. If you understand the layers, you can debug anything: a mispronounced number, a laggy reply, a tool the agent never calls. Each failure lives at a specific layer, and knowing which one saves you days. If you don’t understand the layers, every failure is mysterious and every fix is a guess.
Layer 1: Foundation models
Two families of models can sit at the center of the loop.
Text-first LLMs with vision. GPT-class, Gemini, Claude. They consume text and images and produce text, so for voice you wrap STT and TTS around them (the cascade architecture from Chapter 1). Their strengths are the ones that matter most for complex agents: the best reasoning, the most mature tool calling, and full controllability at every stage of the pipeline. Their weakness is that the wrapping adds latency and integration work.
Realtime, speech-native models. OpenAI’s Realtime API, Gemini Live. They consume and produce audio directly, with no separate transcription or synthesis step. The wins are latency, natural prosody, and the ability to actually hear tone: hesitation, sarcasm, energy. The costs are less mature tool calling, less control mid-pipeline (you cannot inspect or edit a transcript that never exists), and a higher price per minute.
Neither family is “the future” to the exclusion of the other. Choose per use case: a fast-paced conversational companion leans realtime; a tool-heavy, precision-critical workflow often still cascades.
Around the core model sits a supporting cast. STT models must handle accents, noise, crosstalk, and false starts. Test with your real users’ audio, not studio recordings. TTS models differ meaningfully in voice quality and edge-case handling: numbers, formulas, and unusual names trip up some models and not others. A note worth its own sentence: pronunciation problems are usually best solved by picking a different TTS model, not by prompting. Some behaviors live in the training layer, and no instruction reaches them. Finally, image generation models (the fast “nano banana” class) get invoked as tools mid-conversation, which means their generation speed matters as much as their quality.
Layer 2: Media infrastructure
This is the layer builders most often underestimate, so respect what it solves:
- Audio transport over WebRTC, echo cancellation, and device handling across browsers and phones.
- Turn detection: when has the user actually finished speaking? Silence alone is not enough; people pause mid-thought.
- Barge-in: when the user interrupts, the pipeline must cancel TTS mid-utterance and roll back the model’s turn cleanly, without the agent later “remembering” saying things it never finished.
- Session state and reconnection when networks do what networks do.
The key open-source players here are LiveKit and Pipecat. Both are excellent, and both are infrastructure: if you build directly on them, you own the front end, the back end, and the stateful session services yourself. That is weeks of engineering before your first conversation. It is the right call for some teams, a trap for most first products.
Layer 3: Platforms
The platform layer is a spectrum, not a single category.
At the near-infrastructure end sit thin orchestration layers over the media stack: self-hosted pipelines on LiveKit or Pipecat where you still write most of the logic. In the middle sit managed voice-agent platforms (the Vapi and Retell class): you configure prompts, tools, and telephony; they run the pipeline. At the near-application end sit experience platforms like Tough Tongue AI: natural-language configuration, built-in tools (images, quizzes, slides, whiteboards), memory, guardrails, publishing, and embedding.
On Tough Tongue: the platform deliberately spans a wide band of this spectrum. You can use it low-code (describe a scenario in natural language and a builder agent compiles the system prompt, selects tools, and configures memory and guardrails) or high-code, embedding scenarios in your own app and wiring custom tools via HTTP endpoints. Deployment channels include iframe embeds, real phone calls over SIP, and bots that join Google Meet. Part 6 covers the high-code path.
That builder-agent pipeline is worth understanding generically, because most serious platforms converge on it: your requirements go to a meta-prompter that generates the system prompt, selects and configures tools, infers tool data, and sets memory and guardrails, producing a runnable agent composed of a system prompt, a tool registry, a memory manager, guardrails, and a context manager. That second list (guardrails, context management, tool registration) is identical across frameworks. Learn it once and it applies anywhere, which is why Parts 2 through 5 of this playbook teach exactly those components.
Layer 4: Your application
The layer that is always yours, whatever you buy below it: the use case, the domain knowledge, the distribution, and the UX around the conversation. No platform supplies these. A mediocre agent with a sharp use case and real distribution beats a technically brilliant agent aimed at nothing. That is why Chapter 4 is about choosing what to build, and why Part 6 covers wrapping an embedded agent in an application of your own.
Choosing your entry point
A decision guide, by situation:
- You want a prototype this week, and the conversation is the product. Start at the application-layer platform. Descend later if you need to.
- You need deep pipeline control, custom media handling, or on-prem constraints. Build on LiveKit or Pipecat and budget the engineering honestly.
- Your use case is telephony-first (calls, IVR replacement, outbound campaigns). A managed voice platform is the shortest path.
- You have an existing product adding a conversational surface. Pick a platform with a strong embed and API story, because the integration is where you will live.
The honest cost note: platforms charge for convenience; infrastructure charges in engineering time. Both bills are real. For most readers of this playbook, the platform layer is right for version one. Prove the use case at the top of the stack, and descend a layer only when a proven use case demands control the platform cannot give.
The stack tells you what is possible. The next question is what is actually on offer at each layer: the platform landscape, and how to evaluate it.