If you have an AI assistant pointed at your documentation, your customers are already running a retrieval audit on you. They're just not writing the failures down.

Every time someone asks your in-product chatbot a question, every time a developer pastes your docs URL into Claude or ChatGPT, every time a sales prospect runs your product name through a research agent, a silent test happens. A model retrieves some passages from your corpus, composes an answer, and ships it to a human who will either act on it or quietly lose trust. You don't see the score. The user does.

You should run the same audit first, on purpose, with a rubric, before the next support ticket tells you about a failure in your own voice.

What a retrieval audit actually is#

Most teams don't know whether their docs are retrieval-ready because they've never tested it against real queries under real conditions. "Our docs feel fine" is not a signal. It's an absence of one. The retrieval audit is a two-hour exercise that replaces the feeling with a scored, reproducible number, and more importantly, a ranked list of the pages doing the most damage.

A retrieval audit has three parts: a query set, a scoring rubric, and a traceback.

The query set is twenty real questions from real users. Not synthetic. Not the ones you wish people asked. The ones sitting in your support inbox, your community Slack, and your sales-call transcripts. The scoring rubric grades each AI answer on accuracy, completeness, and currency, zero to two each, six to pass. The traceback ties every failure back to the specific page, section, or sentence that caused it.

The output isn't a score. The output is a list of pages ranked by how often they break, and a tag on each explaining why. That list is your remediation backlog.

The twenty-query protocol#

The quality of the audit is the quality of the queries. Synthetic queries produce synthetic confidence. Pull from four sources, five queries each.

Five from your last thirty days of support tickets, specifically the ones that took more than one reply to resolve. First-reply resolutions usually mean the docs worked or the agent pasted a link that worked. Multi-reply threads mean the information was missing, wrong, or hidden, which is exactly what a retriever will also trip over.

Five from your community channels, whether that's Slack, Discord, Discourse, or GitHub Discussions. The ones with no accepted answer are gold. Those are queries where your community couldn't self-serve and your team didn't catch the thread. A retriever has no better shot than they did.

Five from sales-call transcripts, pulled from Gong, Chorus, or whatever you use. Focus on the objections and clarifying questions that came up during demos. These expose the gap between how you talk about the product and how prospects try to understand it, which is also the gap a research agent will fall into when someone evaluates you.

Five from your own onboarding, the questions a new hire asked in their first week. If you don't have a new hire handy, use the questions your last one asked, or ask a friend outside the company to read your homepage and write down what they'd want to know next.

Paste each query verbatim into whatever AI assistant sits on top of your docs. If you have more than one (a Mendable widget, a Claude project with your docs loaded, an internal RAG app), run the set through each and score them separately. A failure on one and a pass on another is itself a finding.

Save the answers. You will need to come back to them.

The scoring rubric#

Three axes, zero to two each, totaling six. Keep the axes narrow so two auditors produce the same number.

Accuracy asks whether anything in the answer is wrong. Zero means a false claim, for example a rate limit that doesn't exist or an endpoint that was renamed. One means technically correct but misleading, like quoting the Pro tier's limits to a user who asked about Free. Two means correct.

Completeness asks whether the answer covers what the user needs to act. Zero means a critical step is missing. One means present but requires a follow-up question, for example "enable the webhook" with no hint of where the toggle lives. Two means the user can close the tab and do the thing.

Currency asks whether the answer reflects the current state of the product. Zero means it references a deprecated feature, old pricing, or a stale endpoint. One means current but undated and unsourced, so the user has to trust you. Two means current and clearly sourced to a dated page the model actually cited.

A score of six is a pass. Four or five is a near-miss worth investigating, usually one axis failing while the others hold. Zero to three is a failure and goes straight to the traceback queue. In our experience running this exercise with SaaS teams, first-run pass rates of forty to sixty percent are typical, and teams that have never thought about retrieval often land below thirty.

The traceback is where the work is#

Scoring tells you that docs are failing. Traceback tells you why. This is the part most teams skip, and it's the part that actually changes the roadmap.

For every failure, open the AI's cited sources, or if your tool doesn't expose them, search the docs for the passages the answer paraphrased. Then ask four questions in order.

Is the right information on the page at all? If no, you have a content gap. Write the missing paragraph. This is the easiest failure to fix and the one teams most often refuse to believe they have.

If yes, is it findable? Is it under a heading the retriever could match on, in a hierarchy that makes sense, with enough surrounding context for the embedding to score well? If not, you have a structure failure. The fix is usually a better H2, a descriptive frontmatter title, or splitting a 4,000-word page into three 1,200-word pages so the right chunk surfaces.

If yes, is it unambiguous? Is there a conflicting page, a deprecated sibling still indexed, a regional override that applies only to EU customers but reads like a global rule? If so, you have a conflict failure. Retrieval pulled the wrong neighbor. Consolidate, redirect, or add explicit scope to both pages (Applies to: US accounts on Pro and above).

If yes, is it self-contained? Or does the paragraph assume context from a parent page the retriever didn't fetch? Chunks are retrieved in isolation. A section that says "as described above" is useless to a model that only got that section. This is a chunk-independence failure, and it's the single most common pattern we see in docs that look fine to humans.

Tag each failure with one of the four modes. That tag tells you who fixes it and how. Content gaps go to the subject-matter expert. Structure failures go to whoever owns information architecture. Conflict failures go to the editor with the delete key. Chunk-independence failures go to whoever can rewrite for context self-sufficiency, which is a skill more than a role.

What to do with the result#

At the end of the audit you have three artifacts: a score, a ranked list of pages by failure count, and a tagged backlog. Treat them in that order.

The score is for the room. Put it on a slide, show it to the people who asked "are our docs AI-ready," and use it as the baseline you'll rerun against. One number, dated, reproducible. If you ran three assistants, show three numbers. Don't average them.

The ranked page list is for triage. In almost every audit, a small number of pages account for most of the failures. Pricing, rate limits, authentication, and whatever your most-churned feature is tend to dominate. Fix the top five pages before you touch anything else. That alone usually moves the next run's pass rate by fifteen to twenty points.

The tagged backlog is for sequencing. Content gaps and conflict failures are cheap and land the same week. Structure failures take a sprint because they touch navigation and redirects. Chunk-independence failures are the longest-lived because they require rewriting for a new reader, the retriever, who arrives without the context a human browser would have. Sequence them that way: gaps, conflicts, structure, then independence.

Rerun the audit every quarter with a refreshed query set pulled from the most recent thirty days. Keep ten queries stable across runs so you can track movement on the same questions, and rotate ten to catch drift as your product changes. The stable ten are your regression suite. The rotating ten are your canary.

Ownership matters. Pick one person who owns the score. Not a committee, not "docs," one name. In most teams, that's the head of docs or developer experience. If no one owns it, the second run won't happen, and the first run becomes a slide deck instead of a practice.

A worked example#

A B2B SaaS team runs the audit on their billing docs. Query seven from the support set reads: "If I downgrade from Pro to Starter mid-cycle, do I get a prorated refund or a credit?"

The AI assistant answers: "Downgrades are prorated and refunded to your original payment method within 5–7 business days."

Accuracy: zero. The actual policy changed six months ago to account credit only, no refunds. Completeness: one, it picks a lane. Currency: zero. Total: one. Hard failure.

Traceback: the model cited /billing/plan-changes, which still carries the old refund language in a paragraph under "What happens when you downgrade." The new policy lives on /billing/credits-and-refunds, published after the change, which the retriever didn't fetch because it ranks lower for the query term "downgrade."

Two failure modes stack here. Conflict, because two pages disagree and both are live. Chunk-independence, because the credits-and-refunds page opens with "As of the October update…" without saying what the update changed, so even if the retriever found it, the chunk wouldn't carry its own authority.

The fix is three lines of work. Delete the refund paragraph on plan-changes and replace it with a one-sentence summary plus a link. Rewrite the opening of credits-and-refunds to name the policy in the first sentence: "Plan downgrades issue account credit, not refunds, effective October 2025." Add frontmatter to both pages with a last_verified date and an applies_to field.

Total time: about twenty minutes. The next audit run, query seven scores six.

Closing#

You don't need a new tool. You don't need a new model. You don't need to wait for your CMS vendor to ship an AI-readiness dashboard. You need twenty questions, a three-axis rubric, and an afternoon.

Your customers are auditing your docs every day. They are not sending you the results. Run the audit yourself this week, write down the number, fix the top five pages, and run it again next quarter. That's the whole practice. Everything else, the vector database, the reranker, the frontmatter schema, is downstream of knowing what is actually breaking.

Do it before your customers do it for you. They already are.