An open census of UK operational surveillance reporting, and what its identifiers actually denote

Two hypotheses we killed before publishing

We began expecting to find that UK operational surveillance outputs carry no persistent identifiers, in contrast with the academic literature where every paper has a DOI. That is false. Every one of the 300 publications we sampled across the three organisations carries a persistent GOV.UK content_id UUID, and none had been withdrawn.

We then expected the report-level identifier field to be unused. Also false. GOV.UK provides a unique_reference field on attachments and it is populated on 38.62 per cent of files. An early probe of a single page returned zero and pointed the wrong way; the full census corrected it.

We report both because a study that only ever confirms its author's starting position is not evidence of anything. What survived those two deaths is sharper than either would have been.

Finding one: the identifier resolves, and denotes the container

A GOV.UK publication page is a mutable container with a stable identifier. A report is a file attached to that container. A recurring series accumulates its editions on one page, so the page identifier stays constant precisely while the thing it denotes changes.

12,769 published files sit under 2,704 page identifiers. 1,161 pages carry more than one file, 253 carry more than ten, and 68 carry more than fifty. The largest single identifier addresses 165 separate files. Across the whole corpus, 10,582 files have no identifier that distinguishes them from their siblings.

The operational consequence is that a reference to a specific edition cannot be made durable. You can cite the page that currently carries the March report. You cannot cite the March report.

Finding two: the recurring series are the least identified

8,758 files announce a date or a period in their own title, which is the evidence that they are editions rather than undated annexes. 5,212 of them, 59.5 per cent, carry no report-level identifier, a worse rate than the corpus as a whole. 242 recurring series carry none on any edition at all.

Finding three: the practice is a habit, not a rule

This is the finding worth acting on, and it is not a technology finding.

The UKHSA National flu and COVID-19 surveillance reports are the weekly national respiratory surveillance output. Each season gets its own page, and the weekly editions accumulate on it. Report-level identifier coverage across consecutive seasons:

Same series, same publishing team, adjacent seasons, and a range of 98.5 percentage points. Nothing about the platform changed. A practice that varies this much between consecutive periods of one series is an individual habit rather than a governed rule, and habits leave when the person holding them does.

That reframes the remedy. This is not a case for new infrastructure. The field already exists, the platform already supports it, and the same team was filling it in at 99 per cent two years ago. It is a case for making one existing practice a documented requirement with an owner.

Finding four: a field named unique_reference is not unique

568 values appear on more than one file. Being fair to the publisher, most of that is legitimate: 362 of those values are format variants, the same report published as a PDF and as a spreadsheet, correctly sharing one reference. That is good practice, not a defect, and counting it as one would have inflated the finding by nearly three times.

What remains after excluding them is real. 129 values denote different documents on different pages, and 75 denote different documents on the same page. The reference ATIC2796 is "Poultry farms with caged systems 2022" on the July 2026 disclosure page and "Poultry statistics 2022" on the June 2026 one.

Our own harvest was wrong, and the gate caught it

The first pass of this census reported 2,718 pages and 12,798 files. Those numbers were wrong. Paging the Search API by descending publication date overlaps when content is republished during the page-through, and 14 pages came back twice.

We did not notice by reading. We noticed because every headline in this study is computed twice, once set-based in Python and once by SPARQL over the graph, with a build that fails when the two disagree. Python counted 12,798 editions and SPARQL counted 12,769, because the duplicated pages produced colliding identifiers in the graph, which the graph correctly merged. The gap was the bug. Deduplication is by page path, and since no identifier was ever shared between two distinct paths, the duplication was ours and not the publisher's. Every figure above is post-correction, and the superseded numbers are recorded in the repository so anyone holding them can see why they moved.

We also had to move the SPARQL side of that gate from rdflib to Oxigraph, because rdflib does not finish the reified two-way joins at this scale. And our own validation engine turned out to be unable to run a SPARQL query inside a batch at all, which we filed against ourselves as the second defect this month found by using our own tooling on our own work.

Being fair to the publishers

These three organisations publish a very large body of surveillance output openly, without a key, under the Open Government Licence, with a persistent identifier on every page and two machine-readable APIs in front of it. That is better than most comparable bodies anywhere in the world, and none of this study would exist otherwise. Every number here is a measurement of transparency, made possible by transparency.

The defect is not that identifiers are absent. It is that the identifier addresses the page while the citable unit is the edition, and that where a report-level identifier does exist, its application depends on who is doing the publishing that year.

We should also state what we did not do. No attachment was downloaded or parsed. Every finding is computed from publisher metadata alone, so the true rate of report identifiers printed inside documents but absent from metadata is unmeasured and could be higher than what we report. The WOAH animal health API refused our requests, so the cross-tier comparison remains unpopulated on that side.

The artefact and an offer

The ontology, the harvest pipeline, the dual-computation gate and an honest build report listing both dead hypotheses and the harvest correction are published at github.com/fabio-rovai/biosurveillance-ontology (opens in new tab). Code MIT, ontology and documentation CC BY 4.0, source data Crown copyright under the Open Government Licence.