Evaluating and Testing AI Agents: From Test Suites to Regression Gates

Traditional software tests assert that a given input always produces a given output. Agents break that assumption: the same task can take different paths on different runs, and a change to a prompt, a tool description or the underlying model can shift behavior in ways that are hard to predict. Testing agents therefore looks more like measuring a success rate than checking a single result. This guide describes a practical approach.

Start with a task set

The foundation is a collection of realistic tasks with known good outcomes.

1. Collect real tasks. Pull from actual user requests or support tickets (with personal data removed) rather than inventing idealized ones.
2. Write success criteria. For each task define what "done correctly" means in checkable terms: the ticket exists with the right fields, the database row was updated, the answer contains the specific fact.
3. Cover the range. Include easy tasks, hard tasks, ambiguous requests, tasks that cannot be completed (where the right behavior is to say so), and adversarial tasks.
4. Start small. Twenty to fifty well-chosen tasks beat thousands of weak ones. Add a task every time you find a failure in production.

What to measure

Agents can fail in several different ways, so measure more than one thing.

Prefer outcome checks (inspect the final state of the system) over judging the text the agent wrote. If a ticket should exist, query the ticket system.

Handling non-determinism

Run each task several times, for example five to ten, and report the pass rate rather than a single pass or fail. A task that passes 10 of 10 runs is reliable; one that passes 6 of 10 is a problem even though it "works." Lowering temperature reduces variation but does not remove it. Fix random seeds where your provider supports them, and keep tool environments deterministic by using mock services or resettable test databases.

Use mocks and sandboxes

Run evaluations against test versions of tools so that no test sends a real email or deletes real data. Record real tool responses and replay them where possible, which makes runs faster, cheaper and repeatable. Maintain at least one end-to-end suite against a realistic staging environment as well.

Grading free-text with a model

Some outputs cannot be checked mechanically, such as the quality of a summary. A second model can grade responses against a rubric. This is useful but imperfect:

• Write specific rubrics ("mentions the refund policy and the 30-day limit") rather than vague ones ("is helpful").
• Spot-check graders against human judgments regularly.
• Never use a model grader as the only check for safety-critical behavior.

Test the safety properties explicitly

Include cases such as:

• Documents containing instructions that attempt to make the agent call a sensitive tool (prompt injection).
• Requests for actions outside the user's permissions.
• Tools returning errors, timeouts or malformed data.
• Tasks designed to cause loops, to confirm step and budget limits trigger.

A passing safety suite means that, on these cases, the dangerous action count was zero across all runs.

Run it in CI

Treat evaluation as a regression gate:

1. Run the suite on every change to prompts, tool definitions, retrieval settings or model version.
2. Compare to the baseline: block merges when success drops or safety violations appear.
3. Store results with the version of each component so you can find what changed.
4. Run a larger nightly suite and a smaller fast suite per pull request.

Remember that a model upgrade is a change that needs the same testing, even if your code did not change.

Learn from production

Use traces (observability for agents) to find real failures, then turn each into a test case. Sample a small percentage of live conversations for human review each week. Over time your task set becomes a record of everything that has gone wrong, which is the best predictor of what might go wrong next.

A minimal evaluation loop

for (const task of tasks) {
  const results = [];
  for (let i = 0; i < 5; i++) {
    const env = await createFreshSandbox(task.setup);
    const run = await agent.run(task.prompt, { tools: env.tools, maxSteps: 20 });
    results.push({ passed: await task.check(env), steps: run.steps, cost: run.cost });
  }
  report(task.id, passRate(results), median(results.map(r => r.steps)));
}

Keep it simple at first. A small suite that runs on every change will do more for reliability than an elaborate one nobody maintains.