Research
What a pass on commerce-v1 does not mean
Our assertions are regular expressions over a final message. That makes them reproducible, and it makes them shallow.
A case is recorded as passed when three conditions hold in order: the response is an answer or a clarification rather than any other response kind, the response is not marked degraded, and every assertion the scenario declares holds against the final message. All three, or the case fails and is written to the artifact with the assertion text that failed, the tools that had been called, and the response as returned. Nothing is discarded and nothing is retried.
Here is the part a benchmark table cannot show you. Those assertions are regular expressions. A message can satisfy them while reasoning badly, and a sound message can fail them by phrasing a correct point in words the scenario author did not anticipate. The supplier fixtures say so in their own source comments: the checks are described there as deliberately incomplete, with an instruction never to equate a string match with task success.
We publish the pass rate anyway, because it is the honest summary of what was automatically checked. It is not a measure of quality and we will not let it be quoted as one. That is also why no number from the benchmark appears anywhere else on this site: pages that discuss capability link to the research section rather than repeating a figure that has a footnote attached to it.
If you think a scenario is wrong, the suite is public and so is the runner. Disagreeing with the rubric in writing is a better introduction to this company than a covering letter.
Read next
See every recorded run, including the failures.
More from FlowFinds
- Newer: Your first sale is funded by us
- Older: Why we built the benchmark before we built the demo
- Everything else is in the news index.
To fact-check anything above before you publish it, write to [email protected].