The mirror problem#

Every week we hear some version of the same complaint: "Our AI assistant is hallucinating. We need a better model." We run the audit. The model is not the problem. The model is doing its job, faithfully summarizing broken inputs.

AI did not cause a documentation crisis. AI exposed one.

That is the uncomfortable version of the argument, and it is the one we are going to make. The conversation about AI-ready documentation, as it is currently held across vendor blogs and conference stages, lets docs teams off the hook. It treats retrieval failure and answer drift as new problems caused by LLMs. They are not new. They are old problems finally being surfaced by a reader that refuses to squint. Your docs have been broken for years. Humans were polite enough not to mention it.

Steelmanning the comfortable story#

Before we take the story apart, we should give it its due. The comfortable story goes like this: LLMs are new. Retrieval is hard. Embedding models behave in ways that are still poorly understood. Nobody could have planned for a world in which a chat interface ingests your entire knowledge base and produces fluent paragraphs. Docs teams are doing their best, and the honest response is patience and investment in new tooling.

Parts of that are true. The interface is new. The composition step is new. Vector search has real quirks that did not exist in keyword-era help centers. Any team working on retrieval quality is working on something legitimately hard.

The story is still wrong in the ways that matter. It grants every docs team a two-year grace period to blame the model. It protects existing content budgets from scrutiny by framing the problem as emergent rather than accumulated. It makes documentation investment look like a response to a new category of tooling, not a correction of prior neglect. And it keeps teams waiting for a model release that will never fix what a model cannot fix.

What the AI is actually doing#

Strip away the branding and an LLM answering from your docs is doing four things in sequence. It retrieves candidate chunks from an index. It ranks them by relevance to the query. It composes an answer from the top results. It formats that answer for the interface.

Three of those four steps have been running against your docs for a decade. Your site search retrieved and ranked. Your support copilot retrieved and ranked. Your IDE autocomplete retrieved and ranked. The only genuinely new step is composition, and composition is what makes silent failure loud.

Consider a concrete case. A developer asks, "How do I rotate an API key?" The retriever pulls three chunks: a 2021 changelog entry announcing key rotation, a 2023 security page describing the old console flow, and a 2024 API reference describing the new endpoint. The ranker weights them by lexical and semantic similarity to the query. All three look relevant. A human reading these pages in three browser tabs would glance at the timestamps, notice the console screenshots look nothing like the current console, and follow the API reference. The LLM has no tabs, no screenshots, and no instinct for which page feels stale. It composes a confident paragraph that mixes the 2023 console steps with the 2024 endpoint name. The answer is wrong in a way that reads right.

The failure was present in every one of those steps before the composer ever ran. Retrieval returned contradictory sources. Ranking could not adjudicate between them. The content itself disagreed with itself. The LLM did not invent the contradiction. It surfaced it in the cleanest possible prose.

The four prior failures AI just made visible#

Every pattern we see in AI-assistant audits maps to a failure mode that predates ChatGPT by a decade. Four of them account for most of what teams are calling hallucination.

Stale content#

Pages go out of date. Nobody reads them often enough to notice. Search keeps returning them because nobody demoted them, nobody archived them, and nobody wired ownership to the product surface they describe. A human reader sniffs the "Last updated: March 2022" line and keeps looking. The AI does not sniff. It reads the page as if it were written this morning and builds an answer on top of it. The content was already lying. The assistant just read it out loud.

Conflicting sources of truth#

Pricing on the marketing site. Pricing in the docs. Pricing in the API reference. Rate limits in the developer portal and different rate limits in the SDK README. Feature availability claimed in a launch post and quietly walked back in a support article. A human confronted with these picks the one that looks canonical, usually the one that matches the URL they trust. The AI picks the one that ranks highest for the query. Those are almost never the same page. This has been breaking customer expectations for years. It broke quietly because humans assumed the cheapest number was a typo. The assistant has no priors about your pricing page.

Implicit prerequisites#

"Enable webhooks first" was assumed because everyone who ever read the page had already been through onboarding. The quickstart referenced a config file that only existed after you ran the CLI. The integration guide assumed you had already generated an API key in a section that lived three clicks away. A new engineer Googled around the gap, found a Stack Overflow thread from 2022, and moved on. The AI cannot Google around the gap. It says what the page says. If the page assumes you are halfway through the task, the assistant answers as if you are halfway through the task, and the user ends up in a state the docs never described.

Page-bound information architecture#

Task-relevant information scattered across four pages, findable only by someone who already knows the structure. Concepts in one section. Configuration in another. Reference in a third. Troubleshooting in a fourth. Humans build this mental model over weeks of using the product and reading the docs in context. The AI gets one query and a retrieval window. It sees whichever fragment ranks, not the structure that gives the fragment meaning. Chunking does not rescue a doc set that was already illegible to anyone who arrived mid-journey.

Each of these patterns predates current LLMs by at least ten years. Each was survivable when the reader was a forgiving human with three tabs open and a coworker on Slack. None is survivable when the reader is a composer that turns fragments into fluent paragraphs and ships them to your customers.

Why the reframe changes the work#

The reframe matters because the remediation is different.

If you believe the problem is new, you invest in prompt engineering, retrieval tuning, reranker experiments, and a procurement cycle for the next vector database. You measure your way through eval suites that grade the model on content the model cannot fix.

If you believe the problem is old, you invest in content architecture, review cadence, canonical sources, explicit prerequisites, and ownership that ties every page to a team that ships the product surface. You measure your way through a doc set that gets smaller, sharper, and easier to retrieve against.

We do the first kind of work too, and retrieval tuning has a real role to play. A well-chunked, well-embedded corpus of sound content outperforms the same content dumped into a naive index. But retrieval tuning on top of contradictory, stale, prerequisite-hiding content produces faster wrong answers. The content investment pays for the retrieval investment. The retrieval investment alone does nothing.

This is the part of the conversation the comfortable story is designed to avoid. Content work is slow, political, and unglamorous. It requires saying no to pages, retiring product lines in writing, and reconciling claims across teams that do not want to be reconciled. Retrieval work can be done by a small team in a quarter. Content work is a program. Teams pick the quarter-long project and wonder why the assistant is still wrong a year later.

The first move#

The teams that accept this framing move faster. They stop waiting for a better model. They run a retrieval audit against their top queries, triage by volume and business impact, rewrite the thirty pages that answer the questions customers actually ask, and repeat. That loop, run four times a year, produces a doc set an LLM can answer from. It also produces a doc set humans prefer.

The teams that reject this framing keep waiting. Every model release makes the problem ten percent smaller and a hundred percent more expensive. The assistant gets better at composing answers from broken sources. The answers get more confident. The failures get harder to spot. The budget grows and the NPS does not move.

If you want a concrete place to start, the first move is unglamorous and cheap. Pull the top fifty queries your users actually ask, in your support tool, your site search, or your assistant logs. For each one, read the answer the assistant produces and the pages it pulled from. Mark which of the four failures is in play. You will not need a fifth category. That list is your backlog. It was your backlog in 2019. Nobody could read it until now.

The mirror is not the problem#

It is tempting to blame the tool. The tool is a mirror. You can argue with the mirror, or you can fix what it is showing you. We know which project is harder. We also know which one pays.

If you want help running that audit and doing the content work that follows, that is the work we do. The reframe is free. The rewrite is the job.