Blog/Research
Evals that survive a model swap
Most eval suites test the prompt, not the job. When the model changes underneath you, the difference shows up the same week.
1 min read · · Boring Agents
- 1Grade the outcome of the job, not the wording of the reply.
- 2Keep a frozen set of cases the builders never look at.
- 3Re-run everything on every model change, not only on a schedule.
A prompt-level test tells you whether a particular string still produces a particular answer. A job-level test tells you whether the certificate got chased, the form got keyed correctly, the buyer got the right flat. Only the second one survives a provider retiring the model you built on.
We keep two sets of cases. The working set is what the team improves against. The frozen set is put away and only used for grading. When the two scores start to drift apart, the agent is learning the test instead of the job, and the frozen set catches it.
None of this is new. It is, however, rarely done for AI running inside small and mid-sized businesses. That's the gap we work in.
Tell us the job your team hates. We'll tell you if AI can do it.
A 30-minute call with the people who would build it. No technical knowledge needed: just describe the job, how often it happens, and how you check it's done right.