commerce-v1 results

How to read this page

Each table below is computed from the recorded artifacts at build time. Runs differ in which scenarios they contain and in which revision of the harness produced them, so a pass rate here is the proportion of attempted cases that passed — not a normalised score, and not comparable across models without first checking that the same scenarios and the same pinned sources were involved. The methodology page sets out what is and is not controlled for.

Across 18 runs there are 51 scored cases, of which 37 passed and 14 did not. 12 cases produced no valid final answer and are counted among the failures.

The resolved-identifier column matters: a requested model name and the identifier that actually served the request are recorded separately, and where they differ the difference is shown rather than smoothed over. Recorded cost is the adapter’s own estimate and is only present for providers that report it, so it is not a cost comparison.

9 scenarios have recorded at least one failure: unit-economics (4/8), conversion-diagnosis (3/4), missing-costs (3/5), memory-budget (3/4), mixed-currency (3/4), supplier-quote (3/4), economics-06-tax-basis (2/4), supplier-minimum-order (5/6), supplier-expired-quote (1/2). The failure text for each is in the raw artifact, and the modes they fall into are analysed in the system cards.

Raw artifacts

18 run artifacts of 18 are published verbatim under /research/commerce-v1/runs/, one JSON file per run. Each contains the measurement timestamp, the requested model and provider, the source digests, the adapter metadata, and for every case the assertions’ outcome, the tools called, the tool diagnostics, the elapsed time, the full response and, where the case failed, the failure text. They are the same files the tables above are computed from.

This build read its figures from the benchmark repository on the build machine. Any run recorded since the published snapshot was taken appears in the tables with no download link, marked not yet published, rather than being hidden.

Model pairings measured so far: openai gpt-5.5, 18 cases, 78% passed, last run 2026-09-05; anthropic claude-fable-5-1, 16 cases, 100% passed, last run 2026-09-05; provider unrecorded gpt-5.5, 14 cases, 43% passed, last run 2026-09-05; anthropic fable, 3 cases, 33% passed, last run 2026-09-05.

Read next

To check these numbers rather than take them, follow the reproduction page: it lists the commands that produced every artifact above. For what the failures were, read the system cards.