Just-in-time knowledge

Consider an agent that trains counselors across five therapy methodologies: CBT and four others. Each methodology has a manual, and each manual runs about fifty pages.

Stuff all five manuals into the system prompt and you have blown any sane token budget several times over, buried the agent’s personality under 250 pages of protocol, and (per Chapter 5) made the agent measurably worse at everything, including the methodologies themselves.

Put the role in the prompt and the manuals in a knowledge base, and the agent pulls the CBT chapter only in the moment a CBT question arises. The other four manuals cost nothing until they’re needed.

That is just-in-time knowledge: the agent knows where to look instead of memorizing the library. It is the third surface of the workspace, and the one that keeps the other two lean.

What goes in the knowledge base

The sorting rule is Chapter 6’s cadence question, applied: needed every turn goes in the prompt; needed sometimes goes in the knowledge base.

Typical knowledge base contents: sales manuals and playbooks. Product catalogs and spec sheets. Procedures and protocols. Policy documents. And per-scenario deep content: case details, or rubrics for twenty scenarios when only one is active per session. All of it shares the same profile: genuinely needed, unpredictably needed, and bulky.

The quantitative smell test from Chapter 5 applies here in reverse: if your system prompt is pushing past 10,000 tokens, something in it almost certainly belongs in the knowledge base. But the qualitative bar governs: must-use-every-turn goes in the prompt; everything else is referential, at any size.

On Tough Tongue: knowledge retrieval is the knowledge_base_search tool: enable it in tools_config, upload documents to the scenario, and the agent queries them mid-conversation. Because it is a tool, everything from Chapter 8 applies: it needs when/interpret/how-often instructions like any other tool, and the tool reference covers the configuration.

The latency budget

Here is the tax that makes voice different from chat: each just-in-time retrieval costs roughly 300 to 500 milliseconds before the agent can answer. In a chat interface, half a second is invisible. In a voice conversation operating inside Chapter 1’s one-second presence budget, it is half the budget, the difference between fluent and hesitant.

Three design responses:

• Retrieve at natural pauses. After the user finishes a long turn. Behind an acknowledgment (“Good question, let me think about that for a second”), which buys the exact pause a human expert would take anyway.
• Front-load predictable retrievals. If the session flow says scenario three is next, the content of scenario three is not unpredictable: the platform or your pipeline can prefetch it before the phase begins.
• Don’t gate every turn on retrieval. Most turns shouldn’t need the knowledge base at all. If they do, your prompt is missing must-use content. That is a filing bug (Chapter 6), not a retrieval problem.

Retrieval styles, and content that retrieves well

Retrieval is not one technique. Three styles, matched to content:

Semantic retrieval (RAG) finds meaning-adjacent chunks. It is the standard for narrative content, tolerant of loose structure. As long as things are written in clear prose and are semantically close to how users ask, it works well.

Keyword and exact search wins for exact terms: SKUs, error codes, function names, part numbers. Semantic similarity is actively unhelpful when the user needs precisely one identifier. Coding agents largely converged on this style for exactly that reason.

List-and-read has the agent list available documents and read whole ones, best when documents are small, self-contained, and well-named. (This is the style behind skill files in coding agents, and Chapter 10 will show it’s a pattern you’ll reuse.)

Now the part that saves the most builders: what retrieves well. The rule of thumb: if your document contains tables, charts, and images, it will probably retrieve poorly. If it’s in story format, you can put a large amount of content in and it will be accurately retrieved.

Practical preparation pass for any document entering the knowledge base:

• Convert tables to sentences: “The Pro plan costs $49 per month and includes unlimited scenarios” retrieves; the same fact in row four of a pricing grid often doesn’t.
• Caption images in words: the retriever reads text, not pixels.
• Break long documents into self-contained sections with descriptive headings, so a retrieved chunk makes sense without its neighbors.

Tip: test retrieval before blaming the agent. Query the knowledge base directly with questions phrased the way a user would ask them, not the way the document is written. If the right chunk doesn’t surface in the top results, no prompt engineering will save the conversation. This ten-minute test should run before every knowledge base ships.

Prompt versus knowledge base is not an either/or decision, though. The best agents split single concerns across both (a little in the prompt, the bulk in the store) and that split is its own pattern, important enough to get the next chapter to itself.