A builder ships a language tutor. Minute five: brilliant (sharp corrections, good pacing, remembers the user’s level). Minute fifteen: still good. Minute twenty-five: it repeats an exercise from minute eight, and then calls the user by the name of an example character from an earlier drill.
Nothing “broke.” No error fired, no limit was hit. The context simply grew until the important early facts (the user’s name, level, and goals) drowned in twenty minutes of drill transcripts. Long conversations don’t fail loudly. They blur.
This chapter is about managing the one part of the workspace that grows without your permission: the conversation history.
The growth problem
Simple arithmetic builds the intuition. A voice conversation produces roughly 150 words per minute. Count both sides plus tool results, and a 20-minute session accumulates thousands of tokens of history, all of it re-read on every turn, per Chapter 6’s amnesiac-employee model.
Two costs compound as it grows. The billing cost grows linearly: every history token is processed again on every new turn. And the attention cost grows with it: Chapter 5’s dilution argument applies within a session just as it does within a prompt. The context window itself rarely overflows in a voice session. Quality degrades long before the hard limit, which is why “we’re nowhere near the token limit” is false comfort.
Summarization: keep the important, forget the wasteful
The rule of thumb: any conversation running past ten minutes needs active summarization. The principle governing it: make sure important things are preserved and wasteful things are forgotten. A bad summarizer loses the user’s name; a good one loses the filler.
What “important” means, concretely, is the list that must survive any summarization:
- User identity and the goals stated at the start
- Decisions and commitments made mid-session
- The current flow position: “completed scenarios 1 to 2; scenario 3 in progress”
- Emotional beats a human would remember: “user got frustrated during the pricing drill”
What is safely forgotten: filler turns, resolved clarifications, the verbatim phrasing of completed exchanges, and tool-call plumbing.
The mechanics are usually platform-handled, but know the shape so you can reason about failures: beyond a threshold, older turns are replaced by a rolling summary while recent turns stay verbatim. The composite the model sees becomes: system prompt, super memory, rolling summary, recent turns, tools. When a late-session agent confabulates about minute three, the rolling summary is where the fact went missing.
Tip: test the blur directly. Run a 25-minute session, then ask the agent about something specific from minute three. If it confabulates, your summarization is dropping the wrong things, or the session is simply too long for one agent, which is the next section’s territory.
When summarization isn’t enough
Summarization has a ceiling, and it arrives around a recognizable threshold: sessions consistently beyond twenty minutes, or sessions with many structurally distinct phases: teach, then quiz, then roleplay, then evaluate.
Notice what’s true about those phases: each one only needs a fraction of the total context. The quiz phase doesn’t need the teaching transcript; the evaluator doesn’t need the icebreaker. At that point, summarizing harder is solving the wrong problem. The right move is to stop asking one agent to carry everything and give each phase its own agent, with exactly the fraction of context that phase needs.
That is sub-agent architecture: one agent, minimal context, one job, then a clean handoff. How to split, what crosses the seams, and who coordinates: that is Part 4.