Research
Commerce Index
What the agent is observing across the catalogue, published on a schedule.
The Commerce Index is a periodic account of what the FlowFinds agent is observed doing when it is asked to reason about commerce. It is a record, not a ranking. Each series below is computed at build time from the recorded run artifacts and the scenario source files; no figure on this page is written by hand, and a series for which no artifact exists is described rather than estimated.
This edition covers observations recorded up to 2026-09-05, drawn from 18 runs holding 51 scored cases, read from the live benchmark repository on the machine that built this page.
Status of these figures
All 18 recorded runs are filed under the status public-development-diagnostic-not-independent-benchmark. They are development diagnostics collected by FlowFinds Solutions on its own harness. They are not an independent benchmark, they were not administered by a third party, and they are not evidence of comparative model quality.
Provenance of this build
Every figure on this page was computed when the site was built, by reading 18 recorded runs containing 51 scored cases, from the benchmark repository present on the build machine. All of them were recorded on 2026-09-05. Nothing on this page is typed by hand; re-running the benchmark and rebuilding is the only way to change it.
Source files published alongside the runs: supplier-scenarios.ts, economics-scenarios.ts, claude-adapter.ts, eval-commerce.ts.
Methodology
The index has one rule, and every other property follows from it: a number appears here only if a file on disk produced it. The pipeline is the same one the rest of the research section uses. At build time it reads each run artifact — a JSON record written by the evaluation harness — and each scenario definition, aggregates them, and hands the result to this page. There is no database between the artifact and the sentence you are reading, and no editorial step in which a figure is chosen.
- Collection. Every scored case in every recorded run is included. Runs are not selected, excluded or reweighted, and failing runs enter the index on the same terms as passing ones.
- Derivation. Counts are counts. Rates are the count divided by the number of attempts recorded for that quantity, with the denominator stated beside it. Where no attempt is recorded the value is reported as not measured rather than as zero.
- Attribution. Each run pins the SHA-256 digest of the source files that produced it; 10 files are pinned across the recorded set, so a change to a scenario cannot silently change the meaning of an older observation.
- Publication. The index is recomputed on every build of this site. An edition is what a build produced, and it is dated by the most recent observation it contains rather than by the day it was published. When artifacts are added, the figures move on the next build without anyone editing this page.
What the index is measuring
The observations are of an agent answering fixed commerce prompts under a harness, not of a market. That is a narrower object than the name might suggest, and the narrowness is deliberate: it is the part of the system whose behaviour can be recorded exactly, replayed, and checked by a reader who disagrees with us. The scope of what is currently recorded is set out in the series below, and the scope of what is not is set out beneath them.
Series A — Composition of the observation corpus
What the agent is asked. 14 scenarios are currently declared across 3 capability groups, carrying 37 assertions in total — a mean of 2.6 per scenario. The composition matters because it determines what the index can see at all.
| Property of the corpus | Count | Share of declared scenarios |
|---|---|---|
| Scenarios declared in source | 14 | 100% |
| Requiring the agent to withhold a claim (a prohibition assertion) | 5 | 36% |
| Carrying a recorded reference answer | 8 | 57% |
| Seeding a prior turn before the prompt | 1 | 7% |
The prohibition share is the one to read closely. It is the fraction of the corpus on which the agent is scored partly for what it declines to assert — a figure it cannot derive from the inputs, or an action it has not actually taken. A corpus without such cases would reward confident invention, so this share is a property of the measuring instrument and not a result.
| Assertion kind | Occurrences | Share of assertions |
|---|---|---|
match | 32 | 86% |
doesNotMatch | 5 | 14% |
Series B — What the agent reaches for
Across the recorded set, 51 of 51 scored cases — 100% — involved at least one tool call before the agent answered. No recorded case was answered from the prompt alone. The distribution below counts every call, and separately the number of cases in which each tool was called at least once, so that a tool called repeatedly within one case is not mistaken for a tool in wide use.
| Tool | Calls | Cases in which it was called | Share of scored cases |
|---|---|---|---|
business | 51 | 51 | 100% |
economics | 16 | 16 | 31% |
supplier | 10 | 10 | 20% |
revenue | 9 | 9 | 18% |
product_evidence | 7 | 7 | 14% |
store_funnel | 6 | 6 | 12% |
supplier_order | 6 | 6 | 12% |
Median time to produce one scored answer, across every case that recorded an elapsed time: 22.0 s. This is harness latency under the recorded conditions and is not a service-level statement about the product.
Series C — Computations the system declined
A refused tool call is not a model failure. It is a guard rail firing: the harness declining to compute on inputs it judges insufficient. These are published because they describe the boundary of what the system will assert, which is harder to see from successes than from refusals.
3 refusals recorded, across 2 tools.
| Tool | Refusals | Reasons recorded, verbatim |
|---|---|---|
economics | 2 |
|
supplier_order | 1 |
|
Series D — How the agent fails, when it fails
Failures are classified by the error the harness recorded, not by inspection of the answer. The taxonomy is mechanical, so a failure cannot be filed under a gentler heading than the evidence supports.
| Mode | What the classification means | Cases | Share of scored cases |
|---|---|---|---|
| Degraded output: no valid final answer | The agent loop returned a response the runner marked degraded, so no final message was available to score. | 12 | 23.5% |
| Required content absent | The answer omitted a value, entity or qualification the scenario requires the model to state. | 1 | 2.0% |
| Prohibited content emitted | The answer contained a string the scenario forbids — typically a figure presented as established when the inputs do not support it. | 1 | 2.0% |
Hardest and easiest observed cases
Ranked by pass rate over every recorded attempt. Attempts per case are small and unequal; these are the extremes of what has been observed, not a difficulty scale.
| Case | Attempts | Passed | Pass rate |
|---|---|---|---|
economics-06-tax-basis | 4 | 2 | 50% |
supplier-expired-quote | 2 | 1 | 50% |
unit-economics | 8 | 4 | 50% |
missing-costs | 5 | 3 | 60% |
conversion-diagnosis | 4 | 3 | 75% |
economics-09-order-budget | 2 | 2 | 100% |
supplier-deadline | 2 | 2 | 100% |
supplier-missing-landed | 2 | 2 | 100% |
Series E — Coverage of the index itself
An index that reported only what it had measured, without reporting what it had not, would be misleading by omission. This series is the index measuring its own gaps.
| Coverage property | Value | What it means for the figures above |
|---|---|---|
| Declared scenarios with no recorded attempt | 0 | These capabilities are defined but unobserved. Every series above is silent about them. |
| Recorded cases no longer declared in source | 0 | Historical observations whose definition has since changed or been withdrawn. They are counted, and they are not re-runnable as recorded. |
| Runs whose artifact omits the provider field | 3 | These cases cannot be attributed to a provider and are grouped under an unrecorded provider rather than assigned to one. |
| Distinct provider and model pairings observed | 4 | Sample sizes differ between pairings, so no comparison between them is offered here. |
| Observation window | 2026-09-05 | Nothing outside this window is described. A short window cannot show a trend, and none is claimed. |
Series defined but not yet published
Two series belong in an index of this name and are absent from this edition. They are named here, with the condition each one requires, so that their absence is legible rather than convenient.
- Observed listing prices across the catalogue. The product records the listing prices it observes when it prices an item. Publishing a distribution of them requires a defined sampling frame — which categories, over which period, with what treatment of duplicates and withdrawn listings — and a cohort large enough that the distribution is not a description of a handful of stores. Neither the frame nor the cohort is settled, so no distribution is published and no figure is asserted.
- Store composition and section ordering. The store engine records the order it chose for a storefront and the reason for each position. Publishing what those choices look like in aggregate requires the same sampling frame, and additionally a treatment of tenant data that keeps individual stores unidentifiable. Until that treatment is written down, nothing about live stores appears in this index.
When either series is published it will appear here computed from artifacts on the same terms as the series above, with its sampling frame stated. It will not appear as an estimate in the meantime.
Limitations
- This is not an independent index. FlowFinds Solutions wrote the scenarios, wrote the harness, ran it, and publishes the result.
- The observations are synthetic. No customer, no real supplier, no real spend and no real order is involved in any recorded run, so nothing here describes a real market.
- Sample sizes are small and unequal. Differences of a few cases between models, tools or scenarios are consistent with chance, and this page deliberately offers no comparison between them.
- The window is short. Nothing in this edition supports a statement about a trend, and none is made.
- A pass is the assertions holding on one attempt under one harness revision. It is not a claim about a model in general.
Checking this edition
Every artifact behind these figures is downloadable in full from the results page, the protocol that produced them is written out on the methodology page, and the commands that regenerate them are on the reproduction page. If a figure on this page is wrong, those three pages are sufficient to demonstrate it. If you would rather learn the reasoning the index is testing than read the index, it is taught in full and for free in learn.