Your search index and your AI pipeline are reading different corpora

GOV.UK does withdrawal properly, and it does not help

When a page is withdrawn from GOV.UK it comes out of the search index completely. We checked: a query filtered to withdrawn documents across the entire 708,433 document estate returns exactly zero results. That is better content discipline than almost any organisation manages on its own intranet.

The withdrawn pages are still there. They sit at their original addresses. They are listed in the public sitemap. The robots.txt file explicitly permits crawling them. And the content API serves them in full, body text intact, to anyone who asks.

We sampled 500 URLs at random from GOV.UK's own sitemap and fetched every one through the content API.

The 95 per cent confidence interval on the withdrawal rate is 4.26 to 8.56 per cent, which puts the extrapolation between roughly 37,000 and 74,000 pages. The oldest withdrawn page in the sample had been withdrawn 12.35 years ago and was still being served.

One concrete case. The page at /guidance/nhs-test-and-trace-how-it-works was withdrawn on 24 February 2022. It does not appear in GOV.UK search. The content API returns 43,671 characters of authoritative-looking public health guidance for it today.

None of this is a criticism of GOV.UK. Keeping withdrawn material addressable is deliberate and defensible for a public record: links in old correspondence still resolve, and the historical record stays intact. The problem appears when a second consumer arrives. The search index was designed for people. A retrieval pipeline is not people. It is built the obvious way, which is to crawl the sitemap, fetch the content endpoint, chunk, embed and serve. Built that way it ingests precisely the content the curated index was careful to exclude.

So the defect is not in the content governance, which is working. The defect is that the curated surface and the crawlable surface are different corpora, and nobody checks which one the AI is eating. If that is true here, consider a corporate estate with no withdrawal workflow at all, where the equivalent of a withdrawn page is a 2019 pricing deck that somebody moved to an archive folder the connector indexes anyway.

Nobody records who is on the hook

We went looking for the metadata that would let an organisation manage this properly. We sampled 300 documents at random and fetched each one in full. Across all of them we observed 133 distinct JSON keys in the response schemas.

Not one of those keys expresses a review date, a next-review date, an expiry, a verification date, a content owner, a maintainer, a steward, or a retention rule. Zero documents out of 300 carried any field of that shape.

This is the finding underneath all the others. The most sophisticated public content platform in the country has no way to record who is accountable for a page being true, or when anyone last checked that it was. Neither does Confluence, SharePoint or Notion out of the box.

Everything downstream follows. You cannot measure freshness against a cadence that was never declared. You cannot route a stale page to an owner who was never named. You cannot tell a retrieval system to prefer the maintained source when nothing in the data says which one that is.

What authoritative actually means

Here is the definition we think the field needs, stated so it can be argued with.

The consequences are uncomfortable and that is the point. An excellent, thorough, carefully written analysis with nobody committed to maintaining it is a working document. A three-line page that a named person re-verifies every quarter is a knowledge asset. Quality does not decide it. Length does not decide it. Where the file lives does not decide it.

Retrieval systems get this exactly backwards. They rank on textual signals, so the well-written working document outranks the maintained stub every single time. Your best analyst's draft beats your maintained policy page, and the assistant presents it with the same confidence.

Note what this makes visible. Commitments do not usually get broken, they get transferred to nobody. In the corpus we studied, 4,810 documents, 8.9 per cent, are owned only by organisations that no longer exist. A 1955 NATO status-of-forces agreement is still attributed to the Foreign and Commonwealth Office, which was merged out of existence in 2020. A further 127 documents are owned by a body GOV.UK itself records as having ceased to exist.

That is why the ontology models a commitment as an object with its own lifecycle rather than as a field on a document. A commitment that lapses is invisible unless lapsing is a state that something can be in.

The Corpus Readiness Index, run on 54,222 documents

Seven dimensions, each computed from a corpus's own metadata and text, with no access to any vendor's index and no cooperation from any vendor required. Run against the GOV.UK guidance corpus on 16 August 2026.

The headline is a geometric mean, deliberately, so that one fatal dimension cannot hide behind six healthy ones. A corpus that is immaculately owned, perfectly deduplicated and entirely locked inside PDF attachments is not 85 per cent useful. It is unusable, and the number should say so.

Freshness of 36.5 is the honest number and it still flatters the corpus, because it measures time since last modification rather than time since last verification. A typo fix resets it without anyone checking a fact. The vivid version comes from GOV.UK's own publishing-era metadata: 13,595 live guidance pages are still tagged to the 2010 to 2015 coalition government, which is more than the 7,946 tagged to the current administration. Twelve are attributed to the 1940 to 1945 Churchill national government and are still served today.

Redundancy looks healthy at 96.9 until you look at the largest cluster, which contains 109 near-identical documents, one per year of an annual guidance series. Annual series are legitimate. The problem is that the year is in the title and nowhere else, so a retrieval system asked what the current rules are picks from 109 candidates on textual similarity alone. One machine-readable date-range field per document collapses that entire ambiguity, and it is the cheapest fix in the whole framework.

Retrieval fitness of 64.5 is where most AI programmes actually die. Nearly a third of the corpus has under 500 characters of indexable text because the answer is inside an attachment. The organisation has documented the answer completely. The retrieval layer sees an opaque blob. No amount of embedding model tuning touches it, because it is a content format problem wearing an AI costume.

A second corpus, and the claim we could not previously prove

Everything above rests on an estate that publishes no verification date, which left the most important claim in the framework as an argument rather than a measurement. We said that time since last modification flatters a corpus, because a typo fix resets it without anyone checking a fact. We could not show it, because GOV.UK does not record the other quantity.

Docs-as-code estates do. Microsoft Learn's house style defines the ms.date field as the date an article was last reviewed for accuracy, and git records independently when the file actually changed. So we scanned two more corpora on 16 August 2026: the Kubernetes documentation, 1,672 documents, and the .NET documentation, 13,412 documents.

Kubernetes names people and never dates a check. Microsoft dates almost every check and rarely names anyone. The two estates fail in opposite directions, and neither manages both across most of its corpus.

On the 13,267 .NET documents carrying both signals, they diverge sharply. The median document was last declared verified 9.38 years ago and last changed 1.94 years ago. Measured by declared verification, 87.8 per cent are stale beyond two years. Measured by modification, 49.6 per cent are. 5,077 documents, 38.3 per cent of the corpus, look current by modification and are stale by their publisher's own verification date.

The obvious objection is that this is just documentation for closed technologies, and it deserved testing rather than dismissing. Splitting the corpus, the frozen legacy subtrees such as WCF and .NET Framework show a 37.4 point gap between the two measures. The actively developed half shows 40.0 points: a median of six months since last change against nearly four years since last declared verification. The effect is larger where the work is live, so it is not an artefact of abandoned content.

What this actually indicts is not Microsoft. Frozen content is a legitimate state, and documentation for a closed technology does not need re-verifying every quarter. The defect is that the corpus cannot say so. A document last verified in 2017 because the technology is closed, one last verified in 2017 because nobody has looked, and one whose date was stamped once and abandoned are indistinguishable in the data. All three present as the same stale date. A declared review cadence and a scope statement are what separate them, which is why both are mandatory fields in the ontology rather than optional ones.

What already exists, and what was missing

We should be clear about what is and is not new here, because two bodies of work sit right next to this one.

ISO 30401 is the formal standard for knowledge management systems, on its 2018 edition with amendments in 2022 and 2024, and currently under revision as ISO/DIS 30401 at the enquiry stage with ISO members. It is a management-system standard with the familiar structure of context, leadership, planning, support, operation, performance evaluation and improvement. It tells you what your management system must do, and it explicitly does not prescribe how to manage knowledge. That is a defensible choice with a consequence worth stating plainly: ISO 30401 says nothing testable about a corpus. No clause defines when a draft becomes authoritative. None requires a review cadence. None defines what makes a document fit to be served as a grounded answer. An organisation can hold the certificate and still fail every dimension in the table above.

K-AI, a document knowledge platform based in Versailles, published a piece on 25 May 2026 arguing that every major AI readiness framework omits unstructured documents, and proposing a Corpus Readiness pillar with six axes: in-document anomalies, cross-document conflicts, divergent duplicates, unmarked obsolescence, traceability in the sense of Article 12 of the EU AI Act, and freshness by segment.

They are right, they published first, and the overlap with what we built is substantial and was arrived at independently. K-AI named the pillar. What their article does not contain is a scoring method, any formulas, an implementation or a measured study, and the axes exist to support a commercial audit rather than something you can run yourself.

So the claim we are making is narrow. The observation that corpus quality is the missing pillar is not ours. What was missing is the part that lets you check it without hiring anyone: a published specification, a runnable instrument, and a real study whose every number regenerates from committed code.

Enforcing it, and why the negative rules are in SHACL

A standard that is not enforced is a preference. The repository ships a SHACL publish gate that runs against a knowledge repository the way a linter runs against a codebase, with a non-zero exit status when a violation is found. Against a worked example seeded with seven specific defects it reports exactly those seven, including a working document marked eligible for the retrieval index, a withdrawn asset still holding a live canonical designation, a superseded page naming no successor, and a retention period with no cited legal basis.

One design note for anyone building something similar. OWL is open-world, so the absence of a maintenance commitment in a graph does not entail that no commitment exists, and no OWL reasoner will ever classify something as a working document by absence. Every rule that matters here is of the form "is there no owner recorded?", which is a closed-world question. The ontology therefore states the necessary condition and the disjointness, and every negative, corpus-scoped judgement is delegated to SHACL. Getting that boundary wrong is the most common way a knowledge governance ontology ends up looking rigorous and enforcing nothing.

The same discipline applies to the measurement. There is no language model anywhere in the counting path. Every number here is deterministic and reproducible. Language models are useful for adjudicating the candidates the scanner surfaces, and they have no business producing the counts.

What to do on Monday

Four things, in order, none of which require a budget.

Find out which corpus your pipeline is actually eating. Ask whoever built your retrieval system whether it consumes a curated index or crawls a surface. If it crawls, ask what it does with content marked archived, superseded or withdrawn. In our experience nobody has asked before.

Count your unowned content. Not content with an empty owner field, but content whose owner is a team that has been reorganised, a person who has left, or a mailbox nobody reads. That number is always larger than anyone expects and it is the cheapest one to produce.

Pick one high-traffic topic and count how many live documents answer it. If the answer is more than one, you have found the mechanism by which your assistant will contradict itself, and you have found it before a customer did.

Stop treating decommissioning as housekeeping. It is the largest single risk reduction available, it is unglamorous, and that is exactly why nobody does it. Give removal its own count on the dashboard or it will not happen.

What we are not claiming

A high Corpus Readiness Index does not mean a good corpus. It means the corpus is measurable and maintained, which is a precondition rather than a guarantee. Scores should not be compared between organisations without comparing the underlying counts and the adapter used; the instrument is designed for longitudinal use on one estate, and for argument, not for a league table.

GOV.UK is close to a best case. It has a dedicated content profession, published standards, an explicit withdrawal workflow and a search index that correctly excludes withdrawn material. These findings are a floor for a typical enterprise estate, not a representative sample of one, and nothing here licenses a claim about what any specific private corpus scores.

The instrument has so far been run against one corpus, which makes it a hypothesis with good production values rather than a settled measurement. Two of the seven findings rest on samples of 300 and 500 documents and are reported with their confidence intervals rather than as corpus facts. The freshness dimension measures the wrong thing and has to, because the right thing is not recorded anywhere. The repository's build report lists every one of these caveats, along with four defects we found in our own code while building it, including a regular expression that silently matched no percentages at all and would have quietly understated a finding had a unit test not caught it. A build report with no mistakes in it is not a build report.

Where this fits

This is the same class of defect we have found in every register and estate examined in this series: a letter O typed in place of a zero inside a Legal Entity Identifier in the European insurance register, checksum-invalid identifiers filed with the SEC across the United States fund universe, and two different misspellings of the word retraction inside the scholarly record. What is not validated is eventually wrong, and what is never measured is never fixed.