A builder spends three weeks evaluating platforms in spreadsheets: feature matrices, pricing tabs, Reddit threads. Then she finally builds the same 20-minute prototype on two of them, and learns more in that afternoon than in the three weeks combined.
The lesson is not that evaluation is useless. It is that most evaluation measures the wrong things. This chapter gives you the five axes that actually mattered, so you can prototype on one or two candidates instead of ten.
The five evaluation axes
1. Modality coverage. Voice quality and latency are the baseline, but the questions that separate platforms are further along: can the agent generate images mid-call? Render structured UI (quiz cards, option cards, slides)? Accept screen share or camera input? Many “voice AI” platforms are voice-only. If your use case needs the classroom picture from Chapter 1, voice-only is disqualifying, and no other feature compensates.
2. Tool ecosystem. What ships built-in: image generation, quizzes, scheduling, knowledge search? Can you register custom tools that call your own endpoints? Is there support for external tool servers (MCP)? The built-in set determines your week-one velocity; the custom-tool story determines whether the agent can ever touch your real systems.
3. Memory support. Session-scoped only, or cross-session? Is it configurable, meaning you can tell the platform what to remember, not just whether to remember? Is it two-tier, with a compact summary plus retrievable detail? (Part 3 explains why that architecture matters; when you see it offered, it is a strong signal the platform has run real long-lived agents.)
4. Context control. Can you see and edit the actual system prompt, or only a description of it? Can you control what enters the context window each turn? Black-box context assembly caps your ceiling at whatever the platform’s defaults achieve. That is fine for simple agents, fatal for the kind this playbook teaches you to build.
5. Embed and export. Public links, iframe and SDK embedding, API access to transcripts and analyses. Your agent will live inside a product eventually. Check that the door is open before you move in.
Mapping the landscape
Picture a horizontal spectrum from raw infrastructure to finished application.
At the left edge: LiveKit and Pipecat, the media infrastructure. You get transport, turn detection, and pipeline primitives; you build everything else. Time to first conversation: weeks.
Next: self-hosted pipeline frameworks built on that infrastructure. You get the loop; you still own hosting, state, and UI. Time to first conversation: days.
Then: managed voice platforms, the Vapi and Retell class. You get a running pipeline, telephony, and configuration surfaces; you still build the application around it. Time to first conversation: hours.
Then: experience platforms, the Tough Tongue AI class. You get the pipeline plus built-in multimodal tools, memory, analysis, publishing, and embedding; you bring the use case and the domain knowledge. Time to first conversation: minutes.
At the right edge: vertical end-applications, someone else’s finished product. You get nothing to build, which is the point and the limitation.
Each step right trades control for speed. The mistake is not being at the “wrong” point. It is being at a point that doesn’t match your constraint. Teams with engineering depth and a pipeline-level differentiator belong left. Teams whose differentiator is the conversation itself belong right.
What transfers, what doesn’t
Platforms will come and go under you. Some of what you build survives the move; some doesn’t. Invest accordingly.
The portable assets, which you should write once and keep in your own repository:
- The system prompt: role, flow, scenarios, principles (Part 2). This is the soul of your agent, and it is plain text.
- Tool instructions: when to use each tool and how to interpret results (Chapter 8).
- Knowledge base content (Chapter 9).
- The memory schema: what to remember about users (Part 3).
- Evaluation transcripts and test cases (Part 7).
The non-portable: pipeline configurations, proprietary tool wiring, platform-specific memory stores. These are the cost of convenience. Minimize the amount of load-bearing logic that lives only there, and lock-in stays manageable. The real lock-in risk lives in proprietary tool formats and memory stores, never in prompts.
Tip: keep a
prompts/folder in version control even when building on a no-code platform. The prompt is source code. Treat it like source code: versioned, reviewed, diffable. Chapter 21 builds a whole iteration discipline on top of this habit.
Cost model basics
Voice agents price per minute, and that minute is composed: STT plus LLM tokens plus TTS plus infrastructure. Realtime speech-native models cost more per minute than cascades. Every image tool call adds a per-call cost. Just-in-time knowledge retrieval adds tokens. None of these is alarming alone; together they compound.
The rough mental model to carry: a 10-minute rich multimodal session costs an order of magnitude more than a 10-minute text chat. Price your product accordingly, and design accordingly too. Part 2’s discipline about what enters the context window is a cost discipline as much as a quality one.
On Tough Tongue: pricing is minutes-based, and the platform exposes usage per scenario and per member through the analytics API, so you can see exactly where minutes go once real users arrive.
Pick per project, not per loyalty. A phone-based appointment setter and an embedded coaching scenario have different ideal stacks, and the evaluation above takes an afternoon of prototyping to run.
You know the machinery and the market. Now the decision that determines success more than any technical choice: what to build.