Research note · Kechun Wu · 2026-11-05

Can you trust an AI earnings note? Checking 1,001 claims against SEC filings

An AI model wrote 50 earnings notes for 10 companies. Plain code then checked every claim against the SEC's structured data and the companies' own press releases. Numbers turned out to be the easy part. Quotes and citations are where AI notes go wrong.

The question

AI research tools can summarize a filing in seconds, but analysts still re-check every number before using it, because language models misstate figures, confuse periods and invent quotes. If a machine could check the note instead, review time could go where it matters. This project measures how far that gets you.

How it works

Inputs. Quarterly figures from the SEC's XBRL company facts (with Q4 derived as full year minus nine months) and the earnings press release (8-K Exhibit 99.1), split into numbered paragraphs.
Writing. A model (claude-sonnet-5-5) writes a four-section note as structured JSON. Every sentence is a claim with a type (numeric, quote or qualitative), a metric, a period, a value and a source.
Checking. Deterministic code recomputes each number from XBRL (within the rounding the sentence uses), requires each quote to appear word for word in the cited paragraph, and requires qualitative claims to cite a real paragraph. Nothing is sent back to the model to fix.

Result 1: the numbers are almost always right

99.6% of 734 numeric claims matched the filings: 494 of 497 (99.4%) were recomputed from XBRL, and 237 of 237 figures with no XBRL equivalent (non-GAAP measures, guidance, customer counts) appear as stated in the press-release paragraph they cite, a weaker check. That held across fiscal calendars (January, April, June and November year-ends as well as December) and for derived fourth quarters (100.0% of 141), the numbers most likely to be wrong in the data layer. The 3 misses are rounding at the edge, such as "21.9%" for a true 21.846%.

Result 2: quotes are the weak spot

Only 85.7% of 70 quotes matched the press release word for word. The typical failure is not an invented quote but a paraphrase dressed up as one ("The CEO described the quarter as strong") or a sentence that stops at "…saying" with the quote left out. Nothing false is stated, but a reader can't see management's actual words, which is the point of a quote. A smaller group of claims cited a paragraph that doesn't contain the number they state.

Result 3: the checker catches errors, but grade it honestly

Corrupting correct claims in known ways (value ±5%, wrong unit, wrong period, swapped QoQ/YoY, one word changed in a quote) and re-checking them caught 100.0% of 1,957 injected errors. That number is close to tautological: it tests the checker's own logic against the same data. The more useful lesson came from reading every flagged claim by hand. Several "errors" were the checker's fault. Sentences were split across HTML blocks, quotes were matched against the wrong field, and cash outflows were read as sign errors, which alone made April fiscal-year companies look worse until it was fixed.

What this means for an analyst

Trust checked numbers, then spend review time elsewhere. With XBRL as ground truth, numeric accuracy is measurable and high.
Read quotes and explanations yourself. About 20% of claims are qualitative and only "sourced", and quotes fail far more often than numbers.
Audit the auditor. A checker that is too strict produces false alarms that look like model errors. Its failures need the same scrutiny as the model's.

Limits

Ten companies and five quarters each is a small sample. The checker treats XBRL as truth, so a company's own tagging error would pass. Whether a qualitative claim is a fair reading of its paragraph is not machine-checkable. A hand-graded sample of 20 claims is in progress as an independent check. No earnings-call transcripts were used.

Data: SEC EDGAR. Code and full results: GitHub. Details by metric and calendar: scorecard.