Open research, July 2026
Provenance beats plausibility, but only if you scope it
The failure that matters when a language model reads a financial filing is not a refusal. It is a confidently wrong number wearing plausible provenance. We built a check that catches those without ever seeing the right answer, scored it against 2,400 human-labelled answers the benchmark already ships, and found two things that do not work as well as the one that does.
Errors recovered
57.4%
of human-labelled wrong answers, no model call
Served accuracy
68.8 to 78.0%
at 60.5% coverage
Same check, whole filing
3.6%
recall, because scope is everything
Gold mismatch found
15 of 51
numeric cases differ by exactly 100x
The supervision set nobody used
FinanceBench is the standard open-book financial question answering benchmark: questions about real filings, each with a gold answer, an evidence string and a page number. Its public release also contains something less discussed. Sixteen model configurations were each run over 150 cases and all 2,400 answers were reviewed by hand and labelled correct, incorrect or a refusal.
That is a labelled training and evaluation set for the separate problem of verifying an answer, and it costs nothing to use because the inference already happened. The distribution is 1,135 correct, 528 incorrect and 737 refusals. Before using it we confirmed we were reading the files as intended by reproducing the paper's headline number exactly: GPT-4-Turbo with a shared vector store scores 19.3% correct here, so 80.7% wrong or refused, against the 81% published.
The verifier, and its contract
The verifier sees the question, the model answer and the source filing. It never sees the gold answer, because in production there is no gold answer. It is fully deterministic: no model is called anywhere in this study.
Results below are on the 1,617 answers that attempted a figure, where 31.2% are labelled incorrect. Served accuracy is the share of unflagged answers a human marked correct, against a 68.8% baseline for serving everything.
| Check | Flag rate | Precision | Recall | Served acc. | Verdict |
|---|---|---|---|---|---|
| C1 against the cited evidence page | 0.395 | 0.455 | 0.574 | 0.780 | the recall instrument |
| C1 against the whole filing | 0.041 | 0.273 | 0.036 | 0.686 | coincidence swamps it |
| C1 with a derivation escape hatch | 0.006 | 0.222 | 0.004 | 0.687 | signal destroyed |
| C2 stated arithmetic does not evaluate | 0.014 | 0.909 | 0.040 | 0.696 | a block rule |
| C3 scale mismatch | 0.004 | 0.000 | 0.000 | 0.686 | null result |
| C3 percentage given as a ratio | 0.000 | n/a | 0.000 | 0.688 | null result |
Two negative results worth more than the positive one
Scope is the whole game. Asking whether the asserted figure appears on the evidence page the case cites recovers 57.4% of labelled errors. Asking whether it appears anywhere in the filing recovers 3.6%. The reason is arithmetic: a filing in this set carries a median of 1,270 distinct numbers against 69 on the cited page, so "the number is in the document" is nearly always true by coincidence. A provenance claim without a scope is not a check.
The obvious safety valve is worse than useless. Many correct financial answers compute a figure rather than quoting one, so the natural refinement is to excuse an answer whose figure is reconstructible by arithmetic from the numbers it cites. Of 638 answers whose figure is absent from the cited evidence, that test excuses 629, or 98.6%. Worse, the excused answers are more often wrong (45.8%) than the ones it cannot excuse (22.2%). Shallow arithmetic reconstruction is anti-informative here, and we report it as a dead end rather than dressing it up.
One check does earn a different role. When a model states arithmetic that does not evaluate, it is wrong 91% of the time, but this fires on only 1.4% of answers. That is a block rule, not a review rule. Both scale and unit checks fired too rarely to support any claim and are reported as null results.
Not an artefact of retrieval mode
An obvious objection is that the provenance check simply detects retrieval failure, so it would score well on configurations with poor retrieval and badly elsewhere. It does not. The check has positive recall in every one of the sixteen configurations, ranging from 0.31 to 0.79, spanning closed book, in-context, oracle and vector-store modes across three model families. Its precision tracks each configuration's own error rate, which is exactly how a fixed-recall detector behaves: 0.93 on a configuration where 78% of answers are wrong, 0.21 where 11% are.
The benchmark also marks an answer correct when its qualitative judgement is right even where its figure differs from gold, so a number-level verifier can only be judged fairly where the gold answer is itself a number. On those 51 cases recall rises to 0.725.
Auditing the benchmark itself
Every evaluation inherits its ground truth. So we checked the 150 open cases against their own artefacts, deterministically.
| Check | Result | Class |
|---|---|---|
| A1 cited snippet appears in its cited page | 189 of 189 pass | clean |
| A2 cited page number locates the text in the PDF | 187 of 187 locatable pages sit at offset exactly +1, none at 0 | defect |
| A6 the two shipped files agree on the gold answer | 15 of 51 disagree, every one by exactly 100x | defect |
| A4 gold honours the unit the question asks for | 4 of 16 percentage questions carry a sub-1 ratio as gold | defect |
| A3 gold figure appears in the cited evidence | 32 of 51 do not, because they are derived metrics | characteristic |
Two findings bite. The page numbers are 0-indexed against the PDF: of 187 evidence pages we could locate by token containment, every single one sits at offset exactly +1 and none at 0. Anyone treating the field as a 1-based page number retrieves the wrong page, every time, which quietly corrupts any retrieval evaluation built on it.
The second is worse. The release carries a gold answer in two places, and for 15 of the 51 numerically comparable cases they disagree, every one by a factor of exactly 100. The dataset file gives a percentage, such as "1.9%"; the results files give the ratio, 0.019. All fifteen are percentage questions. Anyone auto-scoring against the results field inherits a hundredfold unit error on 10% of the subset. The sixteen results files never disagree among themselves, which is what makes the pattern unambiguous rather than noise.
One result is deliberately not filed as a defect. 32 of 51 numeric gold answers do not appear in the cited evidence at any scale, because they are derived metrics: days payable outstanding, fixed asset turnover, multi-year averages. Those cannot appear in a filing by construction. It is a property of the benchmark that bounds what any provenance check can score, and it explains the false positives in ours.
What this means for buyers and builders
- For anyone procuring document AI: accuracy on a benchmark tells you nothing about what happens to the errors. Ask what fraction of wrong answers the system withholds, and at what cost in coverage. Those are the two numbers that determine whether it is safe to put in front of an analyst.
- For builders: a cheap deterministic provenance check, scoped to the retrieved passage rather than the document, buys a nine point lift in the accuracy of what you serve for no inference cost. Put it in before reaching for an LLM judge.
- For evaluation designers: ship one gold answer, not two, and state the indexing convention of every offset field. Both defects here are the kind that silently degrade every downstream result rather than announcing themselves.
- Honest boundary: the verifier is number-centric, so on questions whose answer is a judgement rather than a figure its flag is much less informative. It is a triage instrument, not a correctness oracle, and the precision figures say so plainly.
Open, and reproducible
The numeric layer, the checks, the audit and the scoring harness are public. Every figure on this page regenerates from two commands. No model is called at any point, so a full replication costs nothing but CPU time.
The repository ships code and aggregate results only. It does not redistribute FinanceBench questions, gold answers, model answers or filing PDFs: the upstream release carries no licence file, so no redistribution permission can be assumed, and audit findings cite case identifiers instead so every claim stays checkable. FinanceBench is by Islam and colleagues at Patronus AI, 2023.
Related work: our iXBRL disclosure benchmark measures machine-readable disclosure at scale across UK filings, and the machine-validated ontologies study applies the same evidence-before-assertion discipline to knowledge representation.
