Accuracy scorecard
How often the AI note is right, measured by code against SEC filings, and how often the verifier catches
deliberately injected errors. Computed offline by scripts/scorecard.py; no AI is used to grade.
1001 claims in total: 791 verified, 193 sourced, 13 mismatch, 4 unsupported.
Numeric accuracy
| Quarter type | Claims | Verified % | Mismatch % |
|---|---|---|---|
| Q1-Q3 | 593 | 99.5 | 0.5 |
| Q4 (derived from 10-K) | 141 | 100.0 | 0.0 |
Q4 figures are derived (full year − 9-month YTD from the 10-K), so they are the riskiest numbers in the data layer.
Does the verifier catch errors?
Each verified claim is corrupted in a known way and re-checked. Injections that happen to produce a true statement (e.g. the same value in both periods) are skipped.
| Injected error | Injected | Caught | Caught % |
|---|---|---|---|
| value +5% | 491 | 491 | 100.0 |
| value −5% | 491 | 491 | 100.0 |
| wrong unit | 188 | 188 | 100.0 |
| wrong period | 493 | 493 | 100.0 |
| QoQ/YoY swapped | 234 | 234 | 100.0 |
| quote word changed | 60 | 60 | 100.0 |
Caveat: injection tests check the verifier's own logic (units, periods, rounding, quote matching) against the same XBRL data it uses. They cannot reveal a wrong number in the SEC data itself, or a misleading qualitative claim. That is what the hand-graded sample is for.
Hand-graded sample
Where it fails and why
quote paraphrased, not quoted (10)
The model was asked for a direct quote but wrote a paraphrase ("the CEO described the quarter as strong"). Nothing false is said, but a reader can't see management's actual words, so it doesn't count as a verified quote.
- NOW FY2026 Q3: “The CEO called the quarter an inflection point, saying” · the sentence paraphrases instead of containing the quoted words
- ADBE FY2026 Q1: “The CEO highlighted AI momentum, saying” · the sentence paraphrases instead of containing the quoted words
- ADBE FY2025 Q3: “The CEO highlighted AI momentum, noting that Adobe is the leader in its category.” · the sentence paraphrases instead of containing the quoted words
rounding at the boundary (3)
The model rounded a number the wrong way at the edge, e.g. writing 21.9% when the true value is 21.846%. The error is tiny, but the verifier is strict on purpose: a stated number must be a correct rounding of the filed value. Early notes were partly caused by our own inputs (growth rates sent to the model at 2 decimals, which it then rounded again); inputs now carry 4 decimals.
- SNOW FY2027 Q2: “Sales and marketing expense rose 21.9% year over year.” · stated 21.9 percent, XBRL gives 21.846%
- DDOG FY2026 Q2: “Net income declined 15.3% from the prior quarter.” · stated 15.3 percent, XBRL gives -15.247%
- NDAQ FY2026 Q3: “Operating cash flow was up 135.8% year over year.” · stated 135.8 percent, XBRL gives 135.747%
number not in cited paragraph (2)
A number (often in a qualitative sentence, e.g. "RPO was flat at $4.9 billion") doesn't appear in the paragraph the model cited. The number may even be right, but the citation doesn't support it.
- DDOG FY2025 Q3: “Track growth in larger customers with ARR above $100,000.” · numbers [100000.0] in a qualitative claim are not in P6
- PLTR FY2026 Q3: “Remaining performance obligations were flat sequentially at $4.9 billion, so backlog conversion is a watch item.” · numbers [4.9] in a qualitative claim are not in P19
nothing checkable (bad metric, period or citation) (2)
The claim cites something that doesn't exist, so it can't be checked.
- MSFT FY2026 Q4: “Watch whether surging capital spending continues to pressure free cash flow.” · cited paragraph 'PaymentsToAcquirePropertyPlantAndEquipment' does not exist
- MSFT FY2026 Q3: “Watch whether capital spending, which is far above last year, continues to weigh on free cash flow.” · cited paragraph 'PaymentsToAcquirePropertyPlantAndEquipment' does not exist
Verifier bugs found by reading the results. Reviewing every flagged claim turned up false alarms that were the verifier's fault, not the model's, and were fixed: sentences split across HTML blocks; quotes checked against the value field instead of the words in quotation marks; and cash outflows read as a sign error (which alone made April fiscal-year companies look worse). Each fix has a test, and the injected-error catch rate is re-checked after every fix so the verifier isn't loosened.
What the verifier can't check. Whether a qualitative claim (◦) is a fair reading of its paragraph, and whether the note leaves out something important. Those still need a human reader.