Open research, July 2026
Silence is not assent: what actually catches a wrong AI classification in orbit
AI systems are beginning to file space data under ontology terms automatically. We measured what would catch their mistakes. The domain's best-known ontology can reject almost nothing; the public catalogue can reject almost everything. This is the full study, with the code, the numbers, and the error our own sensitivity analysis caught in it.
The graph
833,403
triples covering all 70,122 catalogued objects
Ontology can refute
2 / 353
classes reachable by its single disjointness axiom
LogMap repair conflicts
0
on 20 candidates, while the catalogue refutes 9
The problem with the standard quality gate
When two vocabularies are aligned, the accepted way to check the result is logical: merge source, target and mappings, then run a reasoner to look for contradictions, and repair whatever produces them. Every method in that family assumes the target ontology contains axioms capable of contradicting something. Disjointness assertions, statements that two kinds of thing can never share a member, are where that power lives.
The Space Situational Awareness Ontology, the reference open ontology of the orbital domain and the version vendored inside a NASA visualisation project, declares exactly one disjointness axiom. It separates Cartesian_Ephemeris from Keplerian_Ephemeris: a guard against confusing two coordinate representations. Nothing separates payloads from debris, operational spacecraft from defunct ones, or geostationary orbits from merely geosynchronous ones.
Propagating that axiom through the subsumption hierarchy reaches 2 of 353 classes: the two formats themselves. One of 62,128 possible class pairs, 0.002 percent, is provably incompatible. For every class an object-level alignment would actually target, no wrong mapping can ever be rejected by reasoning. A clean reasoner report there is silence, not assent. This is not a defect of the ontology; lightweight vocabularies are the norm in operational domains, and our companion measurements across industrial and construction standards find the same regime repeatedly. It is a mis-allocation of evidential burden.
The method: let the catalogue vote
The orbital domain has a compensating asset that few domains can match: a single public catalogue enumerating every tracked object with type, operational status, launch and decay dates, and orbital elements. Where the schema cannot refute, 70,122 instances can. The pipeline below turns that asset into an evaluation channel.
Two design decisions carry the weight. First, the catalogue's codes are lifted into defined classes with stable identifiers and printed thresholds, so the interpretation is inspectable rather than implicit. Second, and more important, instance data never asserts ontology types directly: ontology classification flows only through an explicit alignment file. That keeps the alignment a separate, falsifiable artefact instead of an invisible assumption baked into the data.
Result 1: four plausible wrong mappings, two channels
To compare the channels we authored four correspondences that are wrong but lexically plausible, the characteristic failure mode of lexical, embedding and LLM matchers. Each was scored twice: can any reasoning path reach a contradiction, and how many catalogue-native counter-instances exist.
| Wrong mapping | Why a matcher proposes it | Reasoner | Witnesses | Rate |
|---|---|---|---|---|
| Payload → Operational_Satellite | Payloads are satellites and many are operational | silent | 9,031 / 27,258 | 33.1% |
| GEO band → Geostationary_Orbit | The perennial geosynchronous/geostationary conflation | silent | 1,007 / 1,734 | 58.1% |
| RocketBody → Payload | Near-identical schema features: owner, launch date, elements | silent | 6,870 / 6,870 | 100% |
| Decayed → Resident_Space_Object | Every other class maps under it, so matchers generalise | silent | 35,411 / 35,411 | 100% |
The reasoner is silent on all four, structurally: their targets sit among the 351 unreachable classes. The catalogue refutes all four. The two informative cases are the graded ones: a mapping wrong for a third or a half of its extension is not merely wrong, it is quantifiably wrong, and that quantity is the kind of damage estimate a repair system could minimise against in place of the coherence count it cannot compute here.
Result 2: a real matcher, and what the channel does to it
Authored mutants only model matcher failure, so we ran LogMap 4.0 on the same pair and collected not just its final alignment but its complete candidate pool: the discarded and hard-discarded layers of its lexical index, the high-recall stratum every matcher builds before selection. Twenty distinct candidates.
| Candidate | Pool | Verdict | Conf. | Witnesses |
|---|---|---|---|---|
| Payload → Payload | final | correct | 0.70 | 0 / 27,258 |
| StatusOperational → Operational_Status_of_Spacecraft | final | category error | 0.70 | 16,167 / 16,167 |
| RocketBody → Rocket_Body_Debris | discarded | correct, wrongly discarded | 0.44 | 0 / 6,870 |
| StatusOperational → Satellite_Operator | hard discarded | category error | 0.70 | 16,167 / 16,167 |
| RocketBody → Stellar_Body | hard discarded | absurd | 0.01 | 6,870 / 6,870 |
| Debris → Fragmentation_Debris | hard discarded | the curated trap | 0.31 | no test defined |
Three findings. The vacuity result holds on a live system: none of the 20 candidates has a disjointness-reachable target, and LogMap's coherence-repair stage, its distinctive strength on axiom-rich pairs, emitted an empty conflict set. The repair machinery is not wrong here; it is unemployed. The channel catches what selection let through: half of LogMap's final two-mapping alignment is a category error, mapping the class of operational objects to the ontology's class of operational statuses, refuted by every one of its 16,167 instances, which carry orbital elements no status possesses. And it works as a rescue: the correct rocket-body correspondence LogMap discarded at confidence 0.44 passes with zero witnesses in 6,870 chances. Witness profiles are not only a destructive test; they are a precision-and-recall signal that lexical confidence is not.
Result 3: the method falsified us
An earlier version of this work reported the geosynchronous-geostationary conflation at a 99.6 percent witness rate. That number was an artefact of our own vocabulary. The geosynchronous band had been derived by a cascade that assigned band membership only to objects failing the geostationary test, so band membership already encoded "inclination above 5 degrees" and the measurement was reading the class definition rather than the world.
The threshold sweep caught it: an independent recomputation over all band residents returned 58 percent where the graph returned 99.6. The vocabulary was corrected so the geostationary class nests inside the band, and every number in this study was regenerated. We report it prominently because it is evidence for the thesis at the meta level: extensional measurement is falsifiable in a way that reasoner silence is not, and it falsified us.
| Threshold variant (band ±50 min) | Witness rate |
|---|---|
| Inclination cut-off 1° | 76.4% |
| Inclination cut-off 3° | 66.1% |
| Inclination cut-off 5° (ours) | 58.4% |
| Inclination cut-off 7° | 49.2% |
The rate is stable against the band width (56.0 to 58.9 percent across a four-fold range) because band membership is set by period while the discriminating evidence is inclination. It is sensitive to the inclination cut-off, as it must be: that cut-off is the definition of the disputed concept, so a reader moving it is disputing the concept, not the method. The majority verdict survives the entire plausible range, since station-keeping tolerances are conventionally well under one degree.
Result 4: a model that stays inside the vocabulary
If an ontology cannot reject a wrong term, the practical defence is a model that does not produce wrong terms. We fine-tuned Qwen3-Coder-30B on data derived from this study and published it openly: to our knowledge the first language model targeting a space-domain ontology.
The baseline behaviour is the study's thesis in miniature. Asked for SSAO Turtle, the untuned model invents 13.81 non-existent ontology terms per output, and what it invents is telling: ssao:SpaceObject, ssao:launchDate, ssao:OrbitalElements, fluent camelCase that SSAO never defines. It also declares no prefixes, so nothing it writes parses. Confident, well-formed, unusable, and in this domain nothing would reject it.
| Metric (held-out, n=99) | Base | Tuned |
|---|---|---|
| Turtle parse rate | 0.0% | 98.6% |
| Term conformance (every term real) | 0.0% | 97.2% |
| Hallucinated terms per output | 13.81 | 0.06 |
| Namespace fidelity | 33.3% | 100% |
| Primary class accuracy | 36.1% | 100% |
| Orbit regime accuracy | 0.0% | 80.6% |
| Refusal rate on unanswerable questions | 75% | 100% |
Two honest notes. The domain's canonical error survives: three of fourteen regime errors label a geostationary orbit geosynchronous, though in the safe direction, since every geostationary orbit genuinely is geosynchronous. And the model will volunteer fields you did not supply: given the ISS with no COSPAR number, it emitted the correct 1998-067A from pretraining, which is helpful for a famous object and a fabrication risk for an obscure one. The vocabulary gate catches invented terms; it cannot catch invented values. Both limitations are recorded in the model card rather than left for a user to discover.
The same machinery, as an application
Because the graph carries dates, statuses and derived regimes, the falsification machinery doubles as a rule engine for debris-mitigation reporting. Three rules, each with its assumptions printed:
- 10,016 in-orbit LEO objects launched more than 25 years ago in a non-mission state: a conservative lower bound on the population the IADC 25-year disposal guideline addresses (1,356 non-operational or unknown-status payloads, 636 rocket bodies, 8,024 debris, of 28,238 LEO residents). It is a lower bound because the catalogue records launch dates, not mission-end dates.
- 619 non-operational payloads still in the GEO band rather than raised to the graveyard region: candidates for the super-synchronous disposal the guidelines prescribe.
- 0 rows carrying both a decay date and operational status: a clean negative, and clean negatives belong in reports.
What this means for buyers and builders
- For space AI procurement: a tool "validated against the ontology" has passed a gate that cannot fail. Ask what instance-level checking sits behind the accuracy claim, and ask for witness counts rather than confidence scores.
- For SSA data engineering: stage AI output against instance evidence before it enters the record. The catalogue grades errors quantitatively, which schema validation cannot do at all here.
- For matcher developers: witness profiles are a usable selection signal in both directions, catching final-output errors and rescuing wrongly discarded candidates, and they are continuous rather than binary, which suits the direction OAEI evaluation has been moving.
- Honest boundary: some refusals stay curatorial. The catalogue cannot distinguish fragmentation debris from mission-related debris, so nine of LogMap's twenty candidates have no defined test. The method's value is that it makes explicit which judgements it cannot make for you.
Open, and reproducible
Data snapshot, vocabulary lift, argued alignment, falsification harness, sensitivity sweep, matcher outputs and rule signals are public under CC BY 4.0, with sources pinned by checksum. Every number on this page regenerates from the repository. SATCAT data courtesy of CelesTrak (T.S. Kelso); the SSA Ontology is by Robert J. Rovetto, whose satellite-database work explicitly proposed it as a reference ontology for catalogue sources, the programme this study executes and measures.
Related work: the space debris metrics composition checker measures which orbital-debris indices may legitimately be combined; the industrial and construction crosswalks apply the same falsifiability method to other standards families.
