Checking the knowledge graph your pipeline just built

A pipeline cannot check its own output

This is the asymmetry the whole exercise rests on. An extraction pipeline proposes entities, relations and types, and then writes them out in a formal notation. Every part of that sentence is the same program. If the program has a wrong idea about what a valid identifier looks like, it will write invalid identifiers and then read them back with the same wrong idea, and its tests will pass.

The way out is to bring in a reader that had no part in writing the file. That is not a new idea in software, but it is unusual in graph pipelines, and it is remarkably productive. Every finding on this page came from the same move: parse the output with a second engine, and compare.

Two readers, one file, two different answers

The pipeline’s graph builder defaults an entity’s identifier to its surface text and its type to whatever label the extractor produced. On the documented path, that produces Turtle like this:

<Acme Corp> a <ORG> ;
    semantica:text "Acme Corp" ;
    semantica:confidence 0.91 .

A space is not allowed in an IRI. rdflib parses this anyway: it resolves the subject against the current working directory, hands you file:///home/you/project/Acme%20Corp, and logs a warning that nothing in a production pipeline is reading. Oxigraph, through open-ontologies, refuses the file: Invalid IRI code point ' '.

Both behaviours are defensible readings of the standard’s tolerance rules. Only one of them tells you that your entity identifiers now depend on which directory the batch job happened to run in. If you have ever wondered why two loads of the same corpus produced graphs that would not join, this is a candidate explanation.

The load that succeeds and gives you a twentieth of the data

JSON-LD has a feature that is easy to trigger by accident. A top-level @id sitting beside a top-level @graph does not label the document. It names the graph, which converts every member of that array from a triple in the default graph into a quad in a named one. A plain triple reader keeps the default graph and drops everything else.

Measured on version 0.6.6, a knowledge graph of two entities and one relationship exported to JSON-LD parsed as 2 triples through rdflib.Graph.parse() and 21 quads through Dataset(). The 19 statements in the gap were the content. There is no error, no warning, and no count that looks wrong unless you already know what the right count was.

This is the argument for all-or-nothing loading. A partial load that reports success is worse than a refusal, because every dashboard, every query and every downstream model then computes over a fragment and looks entirely healthy doing it.

Validation that cannot fail

The pipeline generates SHACL shapes from its own ontology, which is a genuinely useful feature. The shapes minted their sh:targetClass values under a /shapes/ path, while the data it generated used a /ns# path. The two never met. No shape selected any node, and pySHACL reported conformance on data that broke every constraint in the file.

A vacuous pass and a real pass produce the same verdict, so any acceptance test written as assert the report conforms would have gone on passing for as long as the project lived. The report needs to say how many focus nodes each shape actually selected, and a shape that selected none needs to be an alarm rather than a silence.

There is a second check that catches this class of defect from the other direction. Closed-world vocabulary checking asks which terms appear in the graph and are declared nowhere. Open-world SHACL cannot ask that question, because in the open world an undeclared term is merely unknown rather than wrong, which is precisely why an extractor that invents a plausible-sounding property sails through it.

The timestamp that deletes rows from your answer

Across the codebase there were 106 timestamps written with datetime.now(), spread over 41 files, and 70 written with the deprecated datetime.utcnow() over 23 more. The first means local time. The second means UTC. Once either has been written into an RDF literal, nothing distinguishes them, and eighteen of the first group were in the export module.

Now run an audit query with a proper bound:

FILTER(?t < "2026-08-19T00:00:00Z"^^xsd:dateTime)

XSD 1.1 makes the comparison between an offset-bearing value and an offset-free one indeterminate, because the offset-free value could fall on either side of the bound by up to fourteen hours. SPARQL raises that as an error, and an error in a filter is not a false. Oxigraph drops the row. The query returns, the shape of the result looks reasonable, and the records it was asked about are the ones missing from it.

The fix upstream tightened the range of the export timestamp to xsd:dateTimeStamp, which requires the offset rather than merely permitting it. That is the version of the type that a schema can enforce.

What was measured

Four things to write into the specification

If you are commissioning a knowledge graph build, each of these corresponds to a defect above that the pipeline’s own test suite had not caught.

1. Read it back with something that did not write it. Acceptance runs the output through an independent, strict parser, and a rejection is a failed acceptance rather than a note for the backlog.
2. Assert on the parsed graph and on counts, never on the file. A serialised string that looks correct can mean nothing. State the expected triple count and check it.
3. Require validation reports to state their coverage. How many focus nodes did each shape select? A shape matching zero nodes is the single most likely reason a validation report is clean.
4. Require stable identifiers and offset-bearing timestamps. Re-run the same input in a fresh process and diff the two graphs. If the identifiers moved, nothing built on them will join, and no amount of downstream cleverness will repair it.

Reproducing this

The probe scripts, their raw output files, the reports as filed, and a verification adapter that runs the strict engine over an export before it reaches disk are all public at semantica-contrib. Everything goes through the pipeline’s documented public API, because a defect reachable only from a private helper is a curiosity, and a defect on the path a user takes is a defect.

A word about the subject. Semantica was chosen because it is good enough to be worth checking, and its maintainer replied to the first report within about an hour, agreed the modelling calls, and asked for the pull request. Reports that arrive with a reproduction, a fix and a test get treated well there, which is why this exercise produced merged code rather than a list of complaints. The method is the transferable part, and it applies to any pipeline that emits RDF, including ones you may already be running.