Foundational Research: Trustworthy Scientific AI
Fluency is saturated. Correctness is not.
Most benchmarks for LLMs on scientific tasks grade the answer with string match or an LLM judge, and both reward a confident, well-formed answer. So a model that emits a plausible-but-nonexistent gene, ontology class or predicate scores as correct. Grade the same output with a closed-world oracle instead, every term has to actually exist, and the picture inverts: models that look equally fluent turn out to be right anywhere from 0% to 100% of the time.
Fluency (raw)
saturated
1.00 for seven of nine models
Verified capability
0.00 – 1.00
on the identical tasks
Oracle
closed-world
deterministic, no LLM judge
The Challenge
Benchmarks that grade on fluency, or ask a bigger model to judge, measure the wrong thing for scientific deployment. A model can write perfectly structured RDF, name a gene, a disease and a relationship, and have half of those names be terms that do not exist in the ontology. In a pipeline that trusts the output, that is not a near miss, it is corrupt data that looks clean. What a scientific evaluation needs is a grader that cannot be satisfied by confident nonsense.
The benchmark
30 real gene-disease facts, each asking a model to write Biolink-typed RDF. The oracle parses the output, checks that every Biolink term it emits actually exists in the Biolink Model, and checks the structure (a real gene, a real disease, a real association predicate). Correctness is set-membership plus constraints, computed deterministically, the same closed-world principle that catches hallucinated terms in generated knowledge graphs, turned into an evaluation. Nine models, five open checkpoints run locally and Claude Haiku, Sonnet and Opus:
| Model | Raw (fluency) | Verified | Term-existence | Fabricated terms |
|---|---|---|---|---|
| Qwen3-Coder-30B-A3B (8bit) | 1.00 | 1.00 | 1.00 | 0 |
| Claude Opus | 1.00 | 1.00 | 1.00 | 0 |
| Claude Haiku | 1.00 | 0.93 | 0.99 | 1 |
| Qwen3.6-35B-A3B (8bit) | 1.00 | 0.77 | 0.92 | 7 |
| Claude Sonnet | 1.00 | 0.73 | 1.00 | 0 |
| Llama-3.2-3B | 1.00 | 0.00 | 0.51 | 49 |
| Qwen2.5-3B | 1.00 | 0.00 | 0.51 | 65 |
| gemma-2-2b | 0.23 | 0.00 | 0.06 | 21 |
| Qwen2.5-0.5B | 0.00 | 0.00 | 0.00 | 0 |
Seven of nine models produce structured Biolink RDF on every task (raw 1.00), so fluency tells you almost nothing. Verified capability spans the full range. Two models tie at the top with perfect verified correctness and zero fabricated terms, one local (Qwen3-Coder-30B) and one frontier (Claude Opus). At the bottom, Qwen2.5-3B and Llama-3.2-3B look identical to the leaders on fluency yet score 0.00 verified, inventing roughly half of every term they emit. The oracle also separates two failure modes fluency grading cannot see: hallucination (the small local models, 49 to 65 fabricated terms) and structural incompleteness (Claude Sonnet, 0.73 verified with zero fabricated terms, its misses are a missing typed disease, not an invented term). You cannot lift the score by writing more fluent RDF, only by using terms that are real.
Two authorities, three task families
The benchmark above uses one ontology and one task. That was the easy case, and it let a local model tie the frontier. So we extended it to a second authority, the Gene Ontology (48,329 terms), and three task families: single-hop gene-disease (Biolink), GO annotation (emit a real GO term for a named process), and cross-ontology multi-hop (gene, disease and process, Biolink and GO in one graph). The oracle now checks term existence across both authorities, and multi-hop cannot be gamed by getting one authority right and inventing the other.
| Model | Overall verified | Biolink | GO annotation | Multi-hop |
|---|---|---|---|---|
| Claude Opus | 1.00 | 1.00 | 1.00 | 1.00 |
| Claude Sonnet | 0.75 | 0.60 | 0.85 | 0.93 |
| Qwen3-Coder-30B | 0.48 | 1.00 | 0.00 | 0.07 |
| Claude Haiku | 0.43 | 0.77 | 0.20 | 0.07 |
| Llama-3.2-3B / Qwen2.5-3B | 0.00 | 0.00 | 0.00 | 0.00 |
On the single-authority task the local Qwen3-Coder-30B tied Claude Opus at 1.00. Add a second ontology and multi-hop, and the tie breaks: Opus scores 1.00 across all three families, the 30B collapses to 0.48, perfect on Biolink but zero on GO annotation, where it fluently emits fabricated 7-digit GO identifiers, and near zero on multi-hop. GO annotation is the discriminator; multi-hop is the hardest. A single family called several models equal; the multi-domain version re-separates them and exposes the cross-ontology hallucination the single family hid.
Why this is the ownable gap
Fluency and correctness are different axes, and only one of them is what you deploy on. The closed-world oracle makes the second measurable without a judge to fool, and it validates in both directions: a genuinely capable model scores 100%, so the benchmark is measuring a real capability the weak models lack, not just failing everything. The method is authority-agnostic, the same oracle runs against GO, ChEBI, Reactome or EDAM, which is the path to a multi-domain suite, and it pairs with the grounded knowledge-graph work that consumes exactly the terms this benchmark checks.
"A model that writes flawless RDF naming genes that do not exist is not almost right, it is confidently wrong in a way a fluency benchmark cannot see. Grade on what exists, not on what reads well, and the leaderboard stops lying to you."
Fabio Rovai, Tesseract Academy
An independent, self-initiated study: the Biolink Model, five locally-run open checkpoints over an OpenAI-compatible endpoint, and Claude Haiku, Sonnet and Opus. Qwen3.6-35B is a reasoning model, scored on its final answer with a large token budget so it can finish reasoning. Two authorities (Biolink and the Gene Ontology) and three task families including cross-ontology multi-hop; ChEBI, Reactome and EDAM extend the same oracle further. The facts are public, so the honest use is relative comparison and a versioned, inspectable oracle, not a secret leaderboard. The Claude models are queried through the claude -p CLI, which loads this machine's configuration, a raw API call would be the cleaner comparison. Full method, per-task outputs and reproducible code in the repository.
Benchmark your own model
The oracle and runner work against any OpenAI-compatible endpoint. Point it at your model.
