What 13 Million Triples Reveal About the Quality of US Federal Vocabularies

Why most ontology quality reports get ignored

Tools that score ontologies have existed for over a decade. They are not much used by the people who publish ontologies, and the reason is not indifference to quality. It is that most reports do not survive contact with a publisher who pushes back.

Three failure modes recur. The first is convention presented as standard. A report says an ontology fails because its classes lack textual definitions. No W3C Recommendation requires textual definitions. The publisher reads one such item, concludes the whole document is somebody's house style, and stops reading, which costs the twenty findings that were real.

The second is inapplicable presented as failed. A SKOS concept scheme gets scored against OWL profile conformance and marked down for carrying no OWL axioms, which it was never meant to carry. The score is now measuring the wrong thing and the publisher is right to ignore it.

The third is an unreproducible subject. A report says the NASA keywords have some number of problems. Which retrieval, on what date, from which of several endpoints? A finding that cannot be re-derived cannot be acted on, and cannot be shown to have been fixed.

The distinction that does the work

Every check in this register declares two things that are usually conflated. The first is normativity, which is a claim about authority. A check is normative only if failing it violates a published specification, and the check must name the clause. Ten of the 26 qualify. Six of those ten are the SKOS integrity conditions S9, S13, S14, S27, S37 and S46, which the SKOS Reference states normatively. The others are RDF syntax conformance, media type correctness, and OWL 2 consistency. One normative check cites government policy rather than a standards body: OMB Memorandum M-15-13 requires federal websites to serve only over HTTPS, which makes transport security a compliance question for these publishers specifically.

The second is severity, which is a claim about consequence, and it is deliberately allowed to diverge from normativity in both directions. An undeclared licence is conventional, because no specification compels a publisher to state a licence inside the payload, and high severity, because a consumer who cannot determine the licence cannot lawfully redistribute derived data. A mislabelled media type is normative, because it violates the media type registration, and low severity, because almost every client sniffs the content anyway.

Findings are expressed as W3C EARL assertions rather than a bespoke result format, because EARL already carries the distinction this domain needs. It has five outcomes, not two, and this register uses them in earnest. A SKOS scheme carrying no OWL axioms is recorded earl:inapplicable for OWL profile conformance, not earl:failed. A reasoner that exhausts its budget yields earl:cantTell, never a silent pass. Of the 728 results in this run, 155 are inapplicable and 50 could not be determined, and publishing those counts is what lets a reader recompute the aggregate under a different weighting.

Assessments attach to a snapshot, never to an abstract asset. Each snapshot records the URL requested, the URL finally resolved to, the HTTP status, the media type actually returned, the byte count and a SHA-256 of the bytes. A publisher can re-fetch and compare hashes. If the hash changed, the finding may be stale and this register is wrong to keep asserting it. That property is what makes the results contestable rather than merely assertive.

An ontology that does not parse, and the 7,782 concepts that depend on it

The Library of Congress MADS/RDF ontology, served at id.loc.gov/ontologies/madsrdf/v1.rdf, fails to parse at line 340. The construct is an rdf:Description node element carrying an rdf:resource attribute, appearing as a child of an owl:unionOf with rdf:parseType="Collection". It occurs 23 times.

The RDF 1.1 XML Syntax grammar is explicit here. The nodeElement production permits only rdf:ID, rdf:nodeID or rdf:about, while rdf:resource is valid on an empty property element. A collection parse type contains a node element list, so its children are node elements. The intended attribute is almost certainly rdf:about.

Two independent parsers agree that something is wrong, and they disagree interestingly about what to do. rdflib refuses the file. OWLAPI, via ROBOT, does not fail: it emits a recognition warning and silently substitutes error entities named Error1 through Error8. The second behaviour is the more dangerous one, because a pipeline built on it produces a graph that looks complete and quietly is not. Silent parser recovery is how a defect survives for years.

The construct appears zero times in the BIBFRAME and PREMIS 3 files, so this is specific to MADS/RDF rather than a pattern across id.loc.gov. It matters beyond one file because the Thesaurus for Graphic Materials, a substantial Library of Congress vocabulary, types its 7,782 concepts with madsrdf:Topic. A live vocabulary is typed against an ontology that no conforming parser will read.

A vocabulary of language codes that carries no language tags

Every one of the 507 concepts in the Library of Congress ISO 639-2 vocabulary violates SKOS integrity condition S14, which states that a resource has no more than one value of skos:prefLabel per language tag.

The cause is not carelessness about labels. It is the opposite. Each concept is given three preferred labels, in English, French and German, which is more multilingual care than most vocabularies manage. None of the three carries a language tag. Three untagged labels are three labels sharing the same absent tag, so the condition is violated 507 times out of 507. The concept for the Banda languages carries "Banda languages", "banda, langues" and "Banda-Sprachen (Ubangi-Sprachen)", and a consumer has no machine-readable way to tell which is which.

This is the class of defect a checker finds in a second and a careful human reader never notices, because on the screen the labels look correct and the multilingual intent is obvious. It is also a one-line fix per label.

The rest of the normative column, and what is conspicuously absent from it

The remaining normative failures are small in number and concrete. Four NASA GCMD concept scheme endpoints return application/xml rather than application/rdf+xml, so a client negotiating content properly cannot tell it has been handed RDF, although the payload is well formed and parses. Three assets carry pairs of concepts asserted both associatively and hierarchically, which SKOS condition S27 makes disjoint: 4,700 of roughly 246,000 pairs in the USGS Common Geographic Areas file, which at that scale looks like a generation artefact rather than editorial slips, 6 of 32 in GCMD Platforms, and 1 of 1,106 in the USGS Thesaurus. Two assets breach S14 on a small proportion of concepts.

What is absent from that column matters as much. There were no failures on retrievability: every asset was obtainable at the address its publisher advertises. There were no failures on HTTPS: every plain HTTP request either redirected to HTTPS or, in the EPA's case, was refused outright, which is stronger. There were no inconsistent ontologies, no unsatisfiable classes, and no violations of the SKOS disjointness conditions S9, S13, S37 or S46 anywhere in the sample. A report that wanted a scandal would have to manufacture one.

The real pattern is governance

The conventional column is where the estate actually shows strain, and it clusters tightly. Twenty-one of 28 assets carry no licence statement anywhere in the payload. Twenty name no publisher or creator. Fourteen carry no version or dated metadata. Thirteen have terms without definitions, and twelve do not return RDF when RDF is requested at their namespace.

None of that breaks a rule, and it would be wrong to describe any of it as an error. It is also the set of properties that determines whether a downstream consumer can act. An organisation building a retrieval pipeline over federal vocabularies has to answer three questions before it can ship: may we redistribute derived data, which version did we ingest, and who do we contact when a term changes underneath us. For most of this sample, the payload answers none of the three, and the answers live in a landing page a machine will not read.

There is also a coverage story worth stating plainly. Several assets sought for this register are not available as RDF at all. NIEM 6.0, the largest US federal data exchange standard, publishes its normative model as XML Schema, and the only RDF-adjacent artefact in its repository is a JSON-LD context of 5,418 bytes. The EPA Substance Registry Services returns HTML when RDF is requested. NIST OSCAL is XML and JSON Schema. Those are recorded as sought and not obtainable rather than scored against criteria they were never built to meet, because a register that lists only what worked is not a register of the estate.

Two mistakes this register made, and why they are on the page

An instrument that audits other people should show its own working, including the parts that were wrong. Two findings from the first run were discarded before publication.

The first was a transport finding attributed to the wrong party. The DCAT-US 3.0 SHACL shapes and the National Archives restrictions vocabulary were retrieved from raw.githubusercontent.com, which serves Turtle files as text/plain. The register booked a media type violation against GSA and the National Archives for a header that GitHub sets. Attributing a mirror's behaviour to a publisher is exactly the category error this whole approach exists to avoid. Those assets are now flagged, and transport checks return inapplicable for them with the reason stated.

The second was a check that measured its own regular expression. The dependency check originally derived namespace prefixes by truncating term URIs at the last slash or hash, then probed the prefixes. It reported http://id.loc.gov/vocabulary/ as a dead dependency of twelve assets. That URL does return 404, but no publisher ever minted it, and every real sub-namespace beneath it resolves normally. The check now probes the referenced terms themselves over a deterministic sample. The corrected count is six assets rather than eighteen, and the original headline was thrown away rather than published.

A third case changed shape rather than disappearing. The NOAA paleoenvironmental thesaurus initially recorded as unparseable. It is served as application/rdf+xml; charset=ISO-8859-1, carries no XML declaration, and contains a byte that is not valid UTF-8 at offset 9,497: the n-with-tilde in "El Nino". A correct HTTP client honours the charset parameter, so the loader now does, and the file parses to 27,326 triples. The underlying issue is real but different from a parse failure, and it is now reported as its own check: detached from its HTTP headers, the file cannot be decoded. Since XML 1.0 expressly permits an external protocol to supply the encoding, it is graded conventional rather than a violation.

Credit where it is due

Reproducible ontology quality tooling is not new, and this register would be dishonest to imply otherwise. The closest prior work is directly on part of this corpus: Mader, Haslhofer and Isaac defined fifteen computable quality functions, implemented them as qSKOS, and ran them over fifteen vocabularies including LCSH, MeSH and NAICS, published at TPDL 2012. They found issues in all fifteen, including 342,848 undocumented concepts in LCSH. The SKOS checks here are a narrower and more conservative descendant of theirs.

OOPS! has catalogued 41 ontology pitfalls, derived from an empirical analysis of more than 693 ontologies, and has been live since 2014. FOOPS! has scored vocabularies against the FAIR principles since 2021. Both come from the Ontology Engineering Group at Universidad Politecnica de Madrid, and between them they are most of the reason automated ontology checking is a solved problem at the level of individual artefacts. The OBO Foundry dashboard goes further and runs a standing, monthly, published report across roughly 190 ontologies, which is the closest thing in the field to what is described here.

What is genuinely new is narrower than a headline would like. It is a standing, contestable register pointed at a national government's estate, that separates specification violations from community conventions, that publishes the snapshot hashes behind every claim, and that gives the assessed publishers a documented right of reply. The OBO dashboard reaches exactly two US federal vocabularies, and only because biomedicine adopted OBO. The last systematic look at federal SKOS vocabularies was fourteen years ago and covered three of them.

What we did not measure

Coverage is a seeded sample, not a census, and it is biased toward agencies that publish RDF at all, which by construction underrepresents the agencies whose semantic assets are least accessible. The Library of Congress accounts for 15 of the 28 assets, so proportions should be read as proportions of this sample, in which one publisher is heavily weighted, and not of the federal estate.

Retrievability was observed once, from one network location, on one day. A transient outage and a permanent removal look identical in a single run, which is the main reason the register is built to be re-run rather than published once. Reasoning is bounded by a time budget, and assets that exhaust it report that they could not be determined, which is a limit of this run rather than a judgement about the asset. Assets typed with MADS/RDF rather than SKOS, such as the Thesaurus for Graphic Materials, fall outside the SKOS battery entirely and are assessed only on retrieval, parsing, documentation and governance.

Every headline figure on this page was computed twice, once set-based in Python over the results and once by SPARQL over the EARL graph, by a script that shares no code between the two paths and exits non-zero if they disagree. They agree on every figure. That is a guard against arithmetic error, not against a wrong idea, and the two discarded findings above are what a guard against wrong ideas looks like.

If this register is wrong about your asset

Publishers have a standing right of reply, and using it is the fastest way to change a result. Three responses are equally welcome. If the finding is wrong, show the retrieval or the check we got wrong, and the register is fixed and re-run, with the correction recorded publicly rather than quietly edited. If the finding is right and you have fixed it, the next run will pick it up, and because snapshots are hashed the improvement is visible as a changed hash and a changed outcome. If the check simply does not apply to your asset, that is the response we most want, because it is a defect in the catalogue rather than in your asset, and the fix is a narrower applicability rule or an outright removal.

Nobody is obliged to agree with a conventional finding. It records a departure from community practice, not a breach of any rule the publisher is bound by, and the register says so on every line.