commerce-v1 methodology

This page is the protocol. It is written so that a reader who distrusts our results can find the exact place where a result could be wrong, and then check it against the artifacts and the sources we publish.

1. Scenario construction

A scenario is a TypeScript object with four parts: an identifier, the prompt text put to the agent verbatim, a reference answer written before any model was run, and a check function containing the assertions. Some scenarios also carry a prior turn, which is appended to the agent’s memory before the prompt so that recall can be tested separately from reasoning.

Scenarios are written to three rules.

• Every number in the prompt is stated in the prompt. No scenario requires outside knowledge, market data or a lookup. The arithmetic is checkable by hand, which is what makes the reference answer authoritative.
• The prompt forbids action. Scenarios that mention suppliers instruct the agent not to order or contact anyone, so that a claim of having done so is unambiguously a failure rather than an instruction followed.
• Under-specification is deliberate. Several scenarios omit a cost, mix two tax bases, or present an expired quote. The correct answer to those is that the figure cannot be determined, and the assertions are written so that producing a confident number fails.

14 scenarios are currently declared, across 3 files. The full inventory with prompts is on the commerce-v1 page.

2. The rubric

Scoring is binary per case. There is no partial credit, no rubric score and no model grading a model. A case passes when the response is an answer or a clarification, is not marked degraded, and satisfies every assertion the scenario declares. Assertions come in two forms: a pattern the final message must match, and a pattern it must not match. The second form carries most of the benchmark’s weight, because it is what stops a plausible-sounding fabrication from scoring.

The complete rubric is reproduced below, scenario by scenario, exactly as it is declared in source. Where a scenario records a reference answer, that is shown too — the reference is the human standard the assertions are an imperfect proxy for.

Core commerce assistant

Unit economics and tax basis

Supplier terms and quote provenance

3. Run procedure

A run executes a selected set of scenarios in sequence and writes one artifact. For each scenario the harness:

1. creates a fresh in-memory store, so no scenario can see another’s state;
2. seeds the prior turn, if the scenario declares one;
3. invokes the agent with the prompt and a tool executor backed by synthetic fixtures, never a live account;
4. records elapsed time, every tool name called, and every tool’s status;
5. applies the assertions and records the outcome, including the failure text;
6. rewrites the artifact after every case, so a run that is interrupted still leaves the cases it completed.

The harness reads COMMERCE_EVAL_CASE, COMMERCE_EVAL_MODEL, COMMERCE_EVAL_PROVIDER and FF_CODEX_ACCOUNTS_DIR from the environment. The case list selects which scenarios run; with no case list the default set runs. Providers accepted are openai and anthropic, and an unknown provider is refused rather than silently defaulted. An empty selection is also refused, so a run cannot succeed by scoring nothing.

4. Source hashing, and why it matters

Before any model is called, the harness computes the SHA-256 digest of each file that can change the result, and writes those digests into the artifact as sourceHashes. The pinned set is:

• lib/commerce/agent.ts
• lib/commerce/economics.ts
• lib/commerce/economics-provenance.ts
• lib/intent/codex-cli.ts
• scripts/eval-commerce.ts
• benchmarks/commerce-v1/claude-adapter.ts
• benchmarks/commerce-v1/supplier-scenarios.ts
• benchmarks/commerce-v1/economics-scenarios.ts
• lib/commerce/supplier-order.ts
• lib/commerce/tools.ts

This covers the agent itself, the economics and tax-basis logic, the supplier-order parser, the tool layer, the model adapter, the scenario definitions and the runner. A result is only meaningful relative to the code that produced it: the same prompt scored against a changed rubric, or answered by a changed agent, is a different measurement wearing the same name. The digests make that detectable rather than arguable. Two artifacts with identical digests were produced by identical code; two artifacts with differing digests are not directly comparable, however similar their numbers look.

Recorded runs pin between 5 and 10 files, because the pinned set itself has grown over time. The union across all runs is 10 files. Every artifact carries its own set, and the digests are in the raw downloads on the results page.

5. How models are adapted

The agent is written against one internal request interface. A model is substituted by supplying a runner that satisfies that interface, so the agent, the tools, the prompts and the scoring are identical whichever model answers. Only the transport differs.

The default provider drives the configured Codex model directly. The Anthropic provider uses a diagnostic adapter that spawns the Claude command line per request, in a fresh temporary working directory that is deleted afterwards. The adapter pins these flags:

The intent of that flag set is isolation: no tools, no MCP servers, no slash commands, no session persistence, no browser, structured JSON output only, and an explicit model identifier that the adapter refuses to run without. The adapter also converts the output schema to the draft-07 subset and throws rather than silently dropping any schema keyword it cannot faithfully transport, so a constraint can never be lost between providers without the run failing. Where the command line reports which model actually served the request, that identifier is recorded in the artifact and shown in the results table — a requested name and a resolved name are not assumed to be the same thing.

Adapter failures are classified into categories — unsupported option, authentication unavailable, requested model unavailable, nested-session restriction, invocation failed — and only the category is recorded. Raw diagnostic output is not written to the artifact, because it can contain credentials and host detail.

6. What is controlled for

• Prompt. Identical text for every model; the prompt is data in a source file, not a template rendered per provider.
• Agent, tools and scoring. Shared code, pinned by digest in every artifact.
• Tool data. Fixed synthetic fixtures. The same call returns the same data in every run.
• State. A fresh memory per scenario; no carry-over, no shared session, no account state.
• Authority. No scenario can place an order, send a message or spend money. The tool layer has no such capability at all.

7. What is not controlled for

• Sampling. Each case is attempted once per run. Model outputs are not deterministic, and no scenario is repeated enough times to separate variance from capability.
• Scenario subset. Runs vary in which cases they include — recorded runs hold between 1 and 6 cases — so two models’ aggregate pass rates may be computed over different scenarios.
• Harness revision. Runs were recorded against several revisions of the pinned sources. Compare the digests before comparing the numbers.
• Reasoning effort and configuration. Effort level, service tier and provider-side routing are set outside the harness and are only partly recorded.
• Provider attribution. 3 early runs predate the provider field and are shown as unrecorded rather than assigned to a provider by inference.
• Cost and latency. Recorded where the adapter reports them, absent otherwise, and measured on a developer machine rather than a controlled environment.

8. Tool refusals are part of the measurement

The fixture tools refuse inputs they cannot honestly compute on, and the refusal is recorded alongside the case. These are guard rails firing, not model errors, and how an agent behaves after a refusal is one of the things the scenarios are designed to expose.

• economics — 2 refusals.
  • Error: The supplied amounts mix tax-inclusive and tax-exclusive bases. Ask for amounts on one consistent tax basis; do not calculate profit or CPA from these values.
  • Error: landed_cost was not supplied in the quoted inputs; use null for an unknown cost.
• supplier_order — 1 refusal.
  • Error: Supply one complete, verbatim founder message containing the supplier terms.

9. Limitations

The following are limitations of the benchmark itself, and they apply to every figure we publish.

• The scoring is shallow by construction. Regular expressions over a final message cannot assess reasoning. They can catch a missing entity, a missing qualification or a forbidden claim. A well-reasoned answer can fail on phrasing, and a poorly reasoned answer can pass by containing the right tokens. The scenario sources describe their own checks as smoke alarms rather than semantic graders, and instruct the reader to inspect the transcript.
• It is first-party. We wrote the scenarios, the assertions and the harness, and we run it on our own machine. The artifacts and sources are published so that this can be audited, not so that it can be waved away.
• The sample is small and unbalanced. With 51 scored cases spread over 4 pairings, most differences between models are within the range that resampling could reverse. No confidence interval is offered because none would be honest at this size.
• The fixtures are synthetic. Real supplier messages are messier than these prompts, real accounting is messier than these fixtures, and performance here does not transfer automatically to production.
• The scenarios cover a narrow slice. They test quantitative and provenance discipline. They do not test product selection, creative work, long-horizon planning, multi-turn negotiation, or anything that requires acting in the world.
• Degraded responses are counted as failures. 12 cases produced no valid final answer. That is the right accounting for a product decision — an answer that never arrives is not a pass — but it conflates model behaviour with transport and adapter reliability.

10. Corrections

If you find an error in a scenario, an assertion, a digest or an artifact, tell us at [email protected]. A confirmed error is corrected at source and the affected runs are re-recorded rather than edited, so the artifact set stays a record of what was measured rather than of what we would prefer to have measured.

Read next

The outcomes this protocol produced are on the results page. To run the protocol yourself, follow the reproduction page. The failure modes the runs revealed are collected in the system cards.

The sources described on this page are published verbatim: supplier-scenarios.ts, economics-scenarios.ts, claude-adapter.ts, eval-commerce.ts.