The Corpus Readiness Index: is your knowledge base fit to feed an AI?

What we built

The repository contains four artefacts, released under CC BY 4.0 and MIT. These include an OWL vocabulary that defines authority as a maintenance commitment with its own lifecycle, ensuring that a lapsed commitment is a state a system can detect rather than an absence nobody sees. It also comprises ten SKOS schemes covering asset status, commitment states, review outcomes and decommissioning routes. A SHACL publish gate runs against a knowledge repository in the same manner a linter runs against a codebase, exiting non-zero in CI when a rule is broken. Finally, the suite includes the Corpus Readiness Index, a written specification with a runnable scanner, which produced the study below. The suite carries 25 passing tests, and a client engagement leaves all four artefacts running inside your estate.

The curated index and the crawlable surface are different corpora

When a page is withdrawn from GOV.UK, it is removed entirely from the search index. We checked: a query filtered to withdrawn documents across the full 708,433 document estate returns exactly zero results. Despite this, the withdrawn pages remain live at their original addresses, are listed in the public sitemap, are permitted by robots.txt, and are served in full by the content API to any request. We sampled 500 URLs at random from the sitemap and fetched every one through the content API.

The 95 per cent confidence interval for the withdrawal rate is 4.26 to 8.56 per cent, suggesting an extrapolation of between roughly 37,000 and 74,000 pages. The oldest withdrawn page in the sample had been withdrawn 12.35 years earlier yet was still being served. A specific example is the page at /guidance/nhs-test-and-trace-how-it-works, which was withdrawn on 24 February 2022 and does not appear in GOV.UK search, while the content API returns 43,671 characters of authoritative-looking public health guidance for it today.

Retaining withdrawn material in a public record is a deliberate and defensible practice, ensuring that links in old correspondence continue to resolve and the historical record remains intact. The search index was designed for human users, yet the retrieval pipeline is built in the standard manner: it crawls the sitemap, fetches the content endpoint, chunks the data, embeds it and serves the results. Consequently, this approach ingests precisely the content that the curated index excludes. The curated surface and the crawlable surface constitute different corpora, and few teams verify which one their assistant consumes. A corporate estate typically lacks a specific withdrawal workflow, so its equivalent is a 2019 pricing deck in an archive folder that the connector indexes regardless.

No field records who is accountable

We then examined the metadata required for an organisation to manage this process. We fetched a random sample of 300 documents in full, observing 133 distinct JSON keys across the response schemas. None of these keys indicated a review date, next-review date, expiry, verification date, content owner, maintainer, steward or retention rule. Zero documents out of the 300 carried any field of that shape.

These downstream processes all depend on this missing information. Without a declared cadence, there is no baseline against which to measure freshness. Similarly, a stale page cannot be assigned to an owner who has not been named. Finally, a retrieval system cannot be instructed to prefer the maintained source when the data does not specify which one that is.

What authoritative actually means

This section presents the definition the field requires, stated so it can be argued with.

A thorough, carefully written analysis that lacks a designated owner for maintenance is a working document, whereas a three-line page re-verified quarterly by a named individual is a knowledge asset. Because retrieval systems rank results on textual signals, the well-written working document often outranks the maintained page, leading the assistant to present both with the same confidence.

Commitments are rarely broken; instead, they are transferred to no one. Within the corpus examined, 4,810 documents, representing 8.9 per cent, are owned solely by organisations that no longer exist, while 127 are held by a body that GOV.UK itself records as having ceased to exist. A 1955 NATO status-of-forces agreement remains attributed to the Foreign and Commonwealth Office, an entity that was merged out of existence in 2020. That is why the ontology models a commitment as an object with its own lifecycle rather than as a field on a document, because a commitment that lapses is invisible unless lapsing is a state that something can occupy.

The Corpus Readiness Index, run on 54,222 documents

The framework comprises seven dimensions, each calculated using the corpus's own metadata and text, requiring neither access to a vendor's index nor cooperation from any vendor. This process was executed against the GOV.UK guidance corpus on 16 August 2026.

The headline metric is a geometric mean, ensuring that one fatal dimension cannot be obscured by six healthy ones. So a corpus that is immaculately owned, perfectly deduplicated and entirely locked inside PDF attachments is not 85 per cent useful, and the score reflects this reality.

A freshness score of 36.5 remains misleading for the corpus because it measures the time since the last modification rather than the time since the last verification, meaning a simple typo correction resets the metric without any factual review. GOV.UK’s own publishing-era metadata reveals the true age of the content: 13,595 live guidance pages remain tagged to the 2010 to 2015 coalition government, a figure exceeding the 7,946 pages tagged to the current administration, while twelve pages attributed to the 1940 to 1945 Churchill national government continue to be served today.

Redundancy at 96.9 appears healthy until the largest cluster is examined, which contains 109 near-identical documents, one for each year of an annual guidance series. While annual series are legitimate, the defect lies in the fact that the year appears in the title and nowhere else. Consequently, a retrieval system asked for the current rules selects from 109 candidates based on textual similarity alone. Adding one machine-readable date-range field per document resolves this ambiguity, representing the cheapest fix in the framework.

Retrieval fitness at 64.5 identifies the specific failure mode that determines the outcome for most AI programmes. A total of 16,131 documents, representing 29.7 per cent of the corpus, contain fewer than 500 characters of indexable text because the substantive answer resides within an attachment. Although the organisation has documented the answer in full, the retrieval layer processes only a title, a brief preamble and an opaque binary object. Adjusting the embedding model does not resolve this issue, as it stems from a content format constraint rather than a retrieval defect.

The second corpus: declared verification against actual change

The GOV.UK study left the framework's most important claim as an argument rather than a measurement: that time since last modification flatters a corpus. GOV.UK records no verification date, so the two quantities could not be compared. Docs-as-code estates record both. Microsoft Learn's house style defines the ms.date field as the date an article was last reviewed for accuracy, and git records independently when the file actually changed. So we scanned two further corpora on 16 August 2026: the Kubernetes documentation, comprising 1,672 documents, and the .NET documentation, comprising 13,412 documents.

Kubernetes names individuals but does not date checks, whereas Microsoft dates nearly every check and rarely names anyone. These two estates fail in opposite directions, and neither achieves both objectives across the majority of their respective corpora.

13,267 .NET documents carry both signals; 5,077 of them (38.3%) look current by modification and are stale by declared verification.

Of the 13,267 .NET documents displaying both signals, the two measures diverge significantly. The median document was last declared verified 9.38 years ago, yet was last changed 1.94 years ago. When assessed by declared verification, 87.8 per cent are stale beyond two years, whereas 49.6 per cent are stale when assessed by modification. That leaves 5,077 documents, representing 38.3 per cent of the corpus, appear current based on modification but remain stale according to their publisher's own verification date.

A natural objection is that this concerns documentation for closed technologies, so we tested it. When the corpus is split, the frozen legacy subtrees, such as WCF and .NET Framework, show a 37.4 point gap between the two staleness measures. The actively developed half shows 40.0 points, with a median of six months since the last change against nearly four years since the last declared verification. The effect is larger where the work is live, indicating that this is not an artefact of abandoned content.

Frozen content is a valid state, and documentation for closed technology does not require quarterly re-verification. The defect lies in the corpus’s inability to distinguish these cases. A document last verified in 2017 because the technology is closed, one last verified in 2017 because no one has examined it, and one whose date was stamped once and then abandoned are indistinguishable within the data. A declared review cadence and a scope statement serve to separate these categories, which is why both are mandatory fields in the ontology.

Prior art: ISO 30401 and K-AI

Two adjacent bodies of work sit next to this one, and the distinction is significant. ISO 30401 is the formal standard for knowledge management systems, currently on its 2018 edition with amendments in 2022 and 2024, and is now under revision as ISO/DIS 30401 at the enquiry stage. It is a management-system standard that uses the harmonised clause structure of context, leadership, planning, support, operation, performance evaluation and improvement. It specifies what the management system must do, but it does not prescribe how to manage knowledge. Consequently, ISO 30401 contains no testable requirements concerning a corpus. No clause defines when a draft becomes authoritative, none requires a specific review cadence, and none defines the criteria for a document to be fit for purpose as a grounded answer. An organisation can therefore hold the certificate while failing every dimension in the table above.

K-AI, a document knowledge platform based in Versailles, published an article on 25 May 2026 arguing that every major AI readiness framework omits unstructured documents. The piece proposes a Corpus Readiness pillar with six axes: in-document anomalies, cross-document conflicts, divergent duplicates, unmarked obsolescence, traceability in the sense of Article 12 of the EU AI Act, and freshness by segment. K-AI published first, and the overlap with what we built is substantial, having been arrived at independently. Their article does not contain a scoring method, formulas, an implementation or a measured study. This work adds that missing element: a published specification, a runnable instrument, and a real study whose every number regenerates from committed code.

The publish gate, and why the negative rules are in SHACL

The repository includes a SHACL publish gate that executes against the knowledge repository and returns a non-zero exit status upon detecting a violation. When applied to a worked example containing seven specific defects, the tool reports exactly those seven issues. These include a working document marked eligible for the retrieval index, a withdrawn asset retaining a live canonical designation, a superseded page with no named successor, and a retention period lacking a cited legal basis.

A design note for those building similar systems is that OWL operates on an open-world assumption, meaning the absence of a maintenance commitment in a graph does not imply that no commitment exists, and no OWL reasoner will classify an item as a working document based on absence. Every relevant rule therefore takes the form of a closed-world question, such as whether an owner is recorded, so the ontology defines the necessary conditions and disjointness, while all negative, corpus-scoped judgements are delegated to SHACL. Misplacing this boundary is the most frequent cause of a knowledge governance ontology appearing rigorous while actually enforcing nothing.

The same discipline extends to measurement. No language model is included in the counting path, ensuring that every number is deterministic and reproducible. Language models serve to adjudicate the candidates identified by the scanner, while the counts themselves are not derived from them.

Four checks to run on your own estate

These four checks, presented in sequence, require no additional budget.

Establish which corpus your pipeline consumes. Establish whether the retrieval system reads a curated index or crawls a surface, and, where it crawls, identify how it processes content marked as archived, superseded or withdrawn.

Count your unowned content. Count the items whose assigned owner is a reorganised team, a departed employee or an inactive mailbox, rather than simply those with an empty owner field. It is the least costly number on this list to produce.

Count the live documents on one high-traffic topic. Select one high-traffic topic and count the live documents that answer it. More than one live answer is the mechanism by which an assistant contradicts itself.

Give decommissioning its own number. Removal represents the largest single risk reduction available in most estates, and an unmeasured activity does not happen.

Scope and method

The Corpus Readiness Index assesses whether a corpus is maintained and measurable, a prerequisite for trusting it as an AI input. These scores are intended for longitudinal application within a single estate, while comparisons between organisations require an examination of the underlying counts and the specific adapter employed.

GOV.UK is close to a best case, supported by a dedicated content profession, published standards, an explicit withdrawal workflow and a search index that correctly excludes withdrawn material. An enterprise estate lacking these disciplines should regard these figures as a minimum baseline. Two of the seven findings rely on random samples of 300 and 500 documents, reported with their confidence intervals. The freshness dimension measures modification because verification dates are not recorded anywhere in the estate, a gap that the ontology's mandatory fields address. The repository's build report details the full method, the caveats and the test suite.

Where this fits

This is the same class of defect we have found in every register and estate examined in this series: a letter O typed in place of a zero inside a Legal Entity Identifier in the European insurance register, checksum-invalid identifiers filed with the SEC across the United States fund universe, and two different misspellings of the word retraction inside the scholarly record. What is not validated is eventually wrong, and what is never measured is never fixed.