At the end of a session on agent building, a participant asks the question everyone is thinking: can the results improve the agent automatically?
The honest answer, worth preserving whole: we are building toward that: a self-improving agent, where the transcript is used to improve the agent itself. But for now, it has to be manual. You see the transcript, you figure out where the model is failing, and accordingly you change the prompt. You need high-quality data to do it well, and figuring out what to pay attention to is the open problem.
That answer contains this entire chapter: the manual loop that works today, the automation being built, and the hard problem sitting between them. Take them in order.
The manual loop, ritualized
This is Chapter 21’s loop, upgraded from self-testing to production mining. Weekly, one hour, non-negotiable. The teams that ritualize it compound; the teams that don’t plateau at whatever launch week produced.
- Sample ten transcripts. Five random, five flagged: early exits, guardrail triggers, unusually short or long sessions. Random catches what you weren’t looking for; flagged catches what you were.
- Tag each failure with a layer label from the Chapter 21 diagnostic table, plus a severity: broke the session, degraded it, or cosmetic.
- Rank. The most frequent session-breaking pattern wins. Resist the pull of the interesting bug over the frequent one: the exotic failure you want to investigate is rarely the one costing you users.
- Fix. One change, tagged version. Chapter 24’s rollback discipline holds forever, not just launch week.
- Verify. Next week’s sample answers whether the pattern receded. If the fix regressed something else, the version log finds the culprit in minutes.
A payoff from Part 4 lands here: if your sessions run on sub-agents, the handoff notes are per-phase summaries the sessions wrote about themselves. Mine the notes first: they localize failures to a phase before you read a single full transcript.
On Tough Tongue: the mining loop runs on the sessions API. Every session carries its transcript, rubric scores, per-topic report card, and extracted variables, and the enriched sessions endpoint filters by date, user, and metadata. Webhooks push
session.analyzedevents to your systems, so the weekly sample can assemble itself.
Transcripts as product research
The bug-hunt framing undersells the asset. The same ten transcripts answer four questions no survey can:
What users actually want. Requests your scenario doesn’t serve yet (“can we practice the follow-up email too?”) are roadmap items delivered verbatim, by people mid-experience, at the exact moment of the gap.
Where users light up. The breakthrough moments (Chapter 18) that land hardest tell you what to double down on. The phases users rush through tell you what to cut. Engagement is legible in a transcript in a way no completion metric captures.
Who your users turn out to be. The personas in real transcripts often differ from Chapter 4’s assumptions, and the memory records your agent writes (Chapter 11’s schema) aggregate into a user-research corpus that no interview program could match for honesty.
What to remember better. Users repeating themselves across sessions is a memory-instruction gap (Chapter 12): invisible in testing, obvious at production scale.
Toward genuine self-improvement
The frontier, mapped honestly rather than pitched.
The blocker is the quality signal. Automating the loop means the system must know which conversations succeeded, and that is the hard problem. A transcript where the user talked a lot might be a great session or a derailed one. Length, sentiment, and completion are proxies, and proxies get gamed the moment anything optimizes against them. “What to pay attention to” is the open question, stated plainly.
What you can do now: define quality per scenario. Chapter 4’s success criterion, operationalized. “The user practiced the pitch aloud twice and received specific feedback” is checkable against a transcript: by a rubric, or by an evaluator model reading sessions against it. Scenario-specific quality definitions are exactly the high-quality data the automation needs. Builders who write them today are building the training signal for their own future automation.
The emerging shape: propose-and-approve. The realistic near-term architecture: an analyzer reads the week’s transcripts against your quality signals, drafts the mining report (the tagging and ranking above), and proposes a prompt change. A human approves. The version log records it. The loop stays human-gated at exactly one point (the judgment call), and everything around that point compounds automatically. When the proposals get good enough, the gate widens. That is the direction of travel, and the weekly ritual above is how you will recognize when it arrives.
The playbook, compressed
One paragraph to carry with you.
Pick a conversation you know how to have (Part 1). Build the workspace, not just the prompt: a structured system prompt, instructed tools, just-in-time knowledge, the hybrid pattern everywhere (Part 2). Give it memory in two tiers (Part 3). When it strains, split it: small agents, clean notes, a supervisor if the flow branches (Part 4). Engineer the friction: resistance, breakthroughs, pressure, guardrails, deliberate opens and closes (Part 5). Iterate on transcripts, encode real expertise, subtract unused tools (Part 6). Launch behind four gates, watch the first twenty sessions, and mine transcripts weekly until the mining automates itself (Part 7).
The models will keep changing. The workspace discipline won’t.
If you want to build as you read, you can start free at Tough Tongue AI, or book time with us to talk through what you’re building.