Methodology
AI answers vary from run to run, so we never claim perfect accuracy. We make three guarantees instead, and show the evidence behind every number.
G1 · Calculation accuracy
Every metric is a pure function of stored runs, computed twice by independent implementations. Any mismatch blocks publication.
G2 · Provenance
Every number links to the runs behind it, with the exact prompt, model version, raw answer and cited URLs.
G3 · Statistical honesty
A number is Verified only after it passes the gate. Below 5 valid runs we show “Insufficient data”, not a percentage.
Data quality badges
- Verified n ≥ 5 valid runs, 95% interval half-width ≤ 7.5 points, canaries passed, no model drift, and the shadow calculator matched.
- Provisional Enough runs to show, but at least one condition failed. The reason is listed.
- Insufficient data Fewer than 5 valid runs. We do not show a percentage.
The runtime gate
- Canaries: control questions with known answers run on every engine every day. A failure quarantines that engine for the cycle.
- Adaptive replication: batches of 5 runs, up to 40 per question and engine (25 on Free). We stop early once the Wilson 95% interval is within ±7.5 points.
- Parse verification: a deterministic parser and an AI judge from a different provider must agree. Disagreements are excluded until a person reviews them.
- Hygiene: empty answers, refusals, errors and duplicates are excluded, and the reason is shown.
- Drift: if an engine changes model version or behaviour, we flag “model changed” instead of reporting a brand change.
- Sealed snapshots: results are stored immutably, with a hash of the input runs and the code version, so any report can be re-verified later.
Measured parser accuracy
Measured on answers labelled by people, not assumed. Before every release the parser must also reach 98% precision and recall on a synthetic test set in CI; that test set is a release gate only and is never presented as accuracy.
No human-labelled evaluation has been published yet. We will publish it here once two people have labelled real engine answers.
Known limitation: a single-word brand name written entirely in lower case is not counted.
Sources
Every external source we use, what it supplies, its license and terms status, and when we last retrieved it: Methodology and sources.
What we never do
We never automate consumer chat apps, never fill in or estimate missing data, and never show synthetic or development data in a customer’s report.