Foundational Research: Ontology-Native AI
The open-world hole: why SHACL cannot catch a hallucinated ontology term
Ask a language model to write RDF and it will, some of the time, give you a term that does not exist: a schema:priceBracket where it meant schema:priceRange, an OBO id off by a digit. The triple parses. It reads correctly. And it is referentially fake. The reflex is to reach for SHACL. Measured on three real vocabularies, SHACL does not catch this. Not some of the time. None of the time.
SHACL false-pass
300 / 300
graphs with a fake term, all pass
Closed-world catch
300 / 300
0 false positives on clean data
Evidence
3 vocabs
418 fabricated terms, reproducible
The Challenge
SHACL is the W3C validation language for RDF, and it is what serious knowledge-graph teams run before they trust generated data. But SHACL is open-world. It validates a data graph against shapes: a shape targets some nodes and constrains some of their properties. If a triple uses a predicate no shape mentions, or an rdf:type no shape targets, SHACL has nothing to say about it, and silence means conformance. It is not asking "does this term exist?". It is asking "do the terms I was told to check satisfy their constraints?". A fabricated term is, by construction, one nobody told it to check. That is the exact failure mode of an LLM authoring RDF, and it falls straight through.
The measurement
We took three real, public vocabularies of very different shapes. For each we generated 100 clean record graphs using only declared terms, and 100 hallucinated graphs identical except for one or two extra fabricated terms, each confirmed absent from the vocabulary but named plausibly enough that a model would emit it. Every graph was validated two ways: with ordinary hand-authored SHACL shapes, and with a closed-world vocabulary gate. The run is deterministic and reproducible in one command.
| Vocabulary | Triples | Classes / props | Fabricated terms | SHACL false-pass | Closed-world catch | False positive |
|---|---|---|---|---|---|---|
| schema.org | 17,949 | 933 / 1,521 | 136 | 100% | 100% | 0% |
| IES4 | 3,976 | 510 / 204 | 134 | 100% | 100% | 0% |
| OBO (PATO+RO) | 270,126 | 2,889 / 759 | 148 | 100% | 100% | 0% |
Across 418 fabricated terms in 300 graphs, open-world SHACL reported conforms=true on every single graph that contained a fabricated term. The closed-world gate flagged one in all 300, and raised zero false alarms on the 300 clean graphs.
See it on one record
A valid-looking schema:Offer with two terms that do not exist in schema.org, the class schema:MerchandiseOffer and the predicate schema:priceBracket:
| Validator | Verdict | Flags |
|---|---|---|
| SHACL (open-world) | conforms = true | none, both fakes admitted |
| Closed-world gate | rejects | MerchandiseOffer, priceBracket |
SHACL conforms because no shape targets those terms, so it never looks at them. This is not a bug in the validator; it is the open-world semantics of SHACL Core working as specified. An LLM that emits a fabricated term therefore sails through the exact tool teams rely on to trust it.
"Just use sh:closed"
The informed objection is that SHACL has sh:closed, which rejects predicates not in an allowed list. It helps, but it is not the guarantee people think. Closed shapes catch extra predicates only if you enumerate every permitted predicate on every shape in advance, do not police the values of rdf:type against the vocabulary, and break the moment a record legitimately uses an imported term you did not list, which in the OBO world (where PATO imports RO imports BFO) is constant. It is per-shape bookkeeping. What generated data needs is a vocabulary guarantee: not "did this node stay inside the property list I wrote for it", but "does every term in this graph denote something the ontology actually defines".
Why the gate is precise, not just strict
The obvious worry about a stricter checker is false positives. The benchmark answers it directly. The gate polices only IRIs whose namespace belongs to the ontology under test, plus namespaces you explicitly name. Standard rdf, rdfs, owl, xsd and sh vocabulary is never flagged, and your own instance identifiers are never flagged, because they are not in a policed namespace. The result is a 0% false-positive rate on 300 clean graphs. It is strict about exactly one thing, whether you used a term the ontology does not define, and silent about everything else. This is the check that ships as a native tool, onto_vocab_check, in our open-ontologies engine, and pairs with our IES4 language model, whose fine-tune cuts hallucinated terms from 0.937 to 0.010 under exactly this kind of closed-world check.
What this does and does not prove
Being honest about scope is the point of publishing the build report alongside the numbers. Three caveats matter. First, the 100%s are a structural property, not a leaderboard score: the gate is defined to catch undeclared terms and the fabricated terms are undeclared, so the contribution is the measurement, at scale, on real vocabularies, that ordinary SHACL practice catches none of them and the gate catches all of them cleanly. Second, this checks term existence, not term appropriateness: a model that uses a real term in a semantically impossible place is a harder problem, and the next gate, certified denotation against a world model, is where that lives. Third, the corpus is generated, not harvested; the natural follow-up is to replay real LLM output through the same gate.
"If you are letting a model write RDF and validating it with SHACL alone, you have an open-world hole, and everything that falls through it looks exactly like clean, conformant data. The fix is one closed-world check, and it costs you nothing on data that was correct to begin with."
Fabio Rovai, Tesseract Academy
This is an independent, self-initiated study built entirely on published open standards and open tools. schema.org, IES4 (Crown Copyright, MIT-licensed), and the OBO Foundry ontologies PATO and RO are used under their open licences. The benchmark, its data and every number above are released open for independent reproduction. Prepared in response to the Encode AI-for-Science Challengescape, backed by ARIA and Pillar VC UK.
Further reading
- SHACL validates shapes, not vocabulary, why the open-world semantics are a feature, and where the companion check belongs.
- The missing box, symbol-existence checking as a named component in a neuro-symbolic pipeline.
- Foundry-grade guarantees for machine-authored ontologies, the same test on the OBO Foundry's own PATO and RO.
- Neuro-symbolic AI has a direction problem, symbols should verify the neural output, not only feed it.
- Beyond existence: certified denotation, the next gate after the vocabulary check.
Reproduce the benchmark
The deterministic script, the three real vocabularies and every number above, in the open-ontologies repository.
