How we grade
A demo works once. A bench shows how often it fails.
A fixed set of your real cases, each with a written right answer, re-run every month for as long as the tool runs. Every agent we take on gets one.
One number, and the list behind it.
This is the format of the scorecard you receive every month. The failures are the useful part: they tell you whether to give the tool more work or less.
Scorecard · shelf checks · October
Pass· graded94 / 100
Failed this month · 6
- 4 photos too dark to read
- 1 duplicate report
- 1 product in a retired category
All six listed, every month. Sample data.
Four steps, no mystery.
- 1
Pull the real cases
A few hundred examples from your own queue: the message half in another language, the form that changed, the invoice with two dates on it. No made-up prompts.
- 2
Define correct in writing
Each case gets an outcome someone will sign their name to. If we can't agree what correct means, that's the finding, and we say so before you spend anything.
- 3
Grade, monthly, forever
The suite re-runs every month and on every model version change. One number, one trend line, one list of what it got wrong.
- 4
Send the scorecard
It goes to you, with the failures listed by case. The failures are the useful part: they tell you whether to give the tool more work or less.
Models change without asking.
Providers ship new model versions on their schedule, not yours. The bench is how we find out what moved, before your customers do. Most AI work is sold on a number nobody can reproduce; this one anyone on your side can check.
Tell us the job your team hates. We'll tell you if AI can do it.
A 30-minute call with the people who would build it. No technical knowledge needed: just describe the job, how often it happens, and how you check it's done right.