Your first eval suite in an afternoon

You do not need a harness to start

Teams often put off evals because they picture a framework, a dashboard and a quarter of engineering time. You can get most of the early value in an afternoon with a spreadsheet, twenty real tickets and a clear idea of what “done” means. This guide walks through that afternoon.

Step one: pull twenty real cases

Open the system your agent will work in and pull twenty closed cases. Do not write them yourself. Invented cases are cleaner than real ones, and the mess is what you are testing.

Pick them on purpose:

• Ten common cases that represent the bulk of the work.
• Five expensive cases where a mistake costs money, trust or a customer.
• Five awkward cases that confused the humans who handled them.

Save the input for each case exactly as the agent will see it, including missing fields and typos.

Step two: write a rubric per case

For each case, write three to five checks. Each check is a yes or no question about the outcome. Avoid checks about tone or wording at this stage. They are hard to grade consistently and rarely the reason an agent fails.

If two people on your team would grade a check differently, it is not a check yet. It is an opinion.

A good set of checks for a refund ticket might read:

1. Refunded the duplicate charge and nothing else.
2. Used the original payment method.
3. Told the customer the amount and the expected date.
4. Did not promise anything the policy does not allow.

Step three: set a pass bar

Decide before you run anything what score is good enough. We suggest two numbers: an overall bar, usually 90% of checks, and a zero-tolerance list for checks that must never fail, such as refunding the wrong account. An agent that scores 97% but fails one zero-tolerance check does not pass.

Step four: run it and read every failure

Run the agent on all twenty cases and grade them by hand. It takes about an hour. Then read each failure and write one line on why it happened. In our experience, failures fall into three groups:

• The agent lacked a tool or a piece of data.
• The policy was ambiguous and the agent guessed.
• The model reasoned badly with everything it needed.

Only the third group is a model problem. The first two are usually faster to fix.

What comes next

Once the agent passes your twenty, grow the set by sampling cases at random, so your own taste stops shaping the test. Automate grading for the checks that are easy to verify. Add shadow mode before anything runs for real. Our post on how we evaluate agents before they touch production covers those later stages.

For today, twenty cases and a rubric will tell you more than any demo.