# Your Agent Needs a Test Suite

# the agent worked. so what?

The first time a coding agent closed a bug I would have spent an afternoon on, it felt like the system was finished. I had a loop. I had tools. I had a result.

Then I swapped the system prompt. The next demo still looked good. Maybe better. Maybe I was just watching a lucky run.

That is the trap. A single success is a demo. Engineering starts when you can tell a real improvement from a vibe.

“Without evals, you're demoing an agent. With evals, you're engineering one.”

share this line

# the whole experiment, not the function

The smallest mental model I trust is four boxes:

A task is one problem. A dataset — or eval set — is a pile of those problems. The agent is the system under test: model, harness, tools, policy. A trial is one attempt at one task. The grader is whatever judges the attempt. The score is what that judge emits. The eval runner is the code that orchestrates the rest.

task

one problem given to the agent

dataset

the collection of those problems

trial

one attempt at one task

grader

the checker you wrote for success

score

what the grader emits

benchmark

a standardized setup used to compare systems

A benchmark is a standardized version of this setup — usually a public dataset plus agreed grading — used to compare systems. Your private repo of nasty bugs can be an eval set without ever being a benchmark.

The first essay in this series was about the agent loop. This one is about the loop you wrap around that loop.

# start with something almost insulting

Before SWE-bench, before judges, before pass@k — a list and a comparison.

const evals = [
  {
    id: "math-001",
    task: "What is 2 + 2?",
    expected: "4",
  },
];

for (const test of evals) {
  const result = await runAgent(test.task);
  const passed = result === test.expected;

  console.log({ id: test.id, passed });
}

This looks too simple to matter. That is the point. Every serious eval system is this loop with better isolation, better graders, and more honest aggregation.

runAgent() is not the eval. It is the thing you are evaluating. Confuse the two and you will start “improving” the test until the agent looks good.

# a grader is just the checker

Pull the comparison out so it has a name.

function grade(result: string, expected: string) {
  return result === expected;
}

const result = await runAgent(test.task);
const passed = grade(result, test.expected);

There is no universal grader(). You design the checker from what success means for that task. “Equals 4” is a grader. “The test suite exits 0” is a grader. “A human reviewer says the tone is fine” is a grader. Same job, different instruments.

# coding agents do not return a string

A math question ends in an answer. A coding agent ends in a repository that used to be broken.

{
  id: "auth-001",
  task: "Fix the authentication bug",
  repo: "./repos/auth-001",
}

The agent receives a task and a checkout. It explores, edits, runs commands, maybe opens a browser. When it stops, the interesting artifact is the working tree — not the last chat message.

So we write a checker for the repository:

async function runTests(repoPath: string) {
  const { exitCode } = await exec("npm test", { cwd: repoPath });
  return exitCode === 0;
}

const result = await runAgent(test.task);
const testsPassed = await runTests(test.repo);
const score = testsPassed ? 1 : 0;

Hold the two functions apart:

runAgent()   // performs the task
runTests()   // grades the resulting repo

runTests() is not a built-in “AI evals” primitive. It is a grader we chose because software already has a culture of executable proof. Lint, types, and a screenshot pass can sit next to it. None of them are the agent.

# the coding-agent eval loop

Scale the insulting loop to a folder of repos.

for (const test of evals) {
  await resetRepo(test.repo);
  await runAgent(test.task, { cwd: test.repo });

  const passed = await runTests(test.repo);
  const score = passed ? 1 : 0;

  console.log({ id: test.id, score });
}

Two hundred tasks later you are not staring at a vibe. You have a rate:

78 / 100 tasks passed
= 78% success rate

That number is only as honest as the isolation around each trial. If task 47 leaves a dirty node_modules, a leaked env var, or a git stash that task 48 can read, you are no longer measuring the agent.

# where the problems come from

You do not invent runTests() in a vacuum. The dataset is the pile of real problems. The grader is how you verify a solution.

task / issue description
repository snapshot
environment / setup
verification criteria

SWE-bench is the canonical public version of this idea: real GitHub issues, real repositories, and a patch graded by whether it makes the right tests pass. The original set is 2,294 Python issues. SWE-bench Verified is a 500-task subset that humans filtered so a fail is more likely to mean “the agent missed” than “the test was nonsense.”

That is also why leaderboards move so fast they become a little dangerous. SWE-bench Verified is approaching saturation for frontier systems. A one-point jump on a nearly-solved set is a weaker signal than it looks.

# you design the grader for the job

Unit tests are one instrument. They are not the category.

Code-based graders are cheap and boring in the best way: exact match, regex, unit tests, integration tests, lint, types, static analysis, “does this row exist in the database.” When the outcome is machine-checkable, start here.

Model-based graders exist because some outputs have no single string. Summaries, tone, “did it actually answer the question,” code taste. The shape is still a function:

const score = await judge({
  task,
  answer,
  rubric,
});

A judge without a rubric is another vibe. A judge that never gets checked against humans will drift. Give it an escape hatch — unknown beats a confident hallucination.

Human graders stay in the loop for the cases automation cannot own: taste, safety, legal, “would I merge this.” Serious systems mix all three. Anthropic's Demystifying evals for AI agents is the clearest public write-up of that mix I have read.

# the agent produces a trajectory

A completion model is input → output. An agent is a trace:

1. 01

   task — Fix the empty-password login path.

2. 02

   model — Proposes reading auth middleware and the failing test.

3. 03

   tool — read / grep / edit — mediated by the harness.

4. 04

   result — File contents and command output come back.

5. 05

   model — Uses the observation to choose the next action.

6. 06

   final — A patched repo — not just a sentence.

People call this a trajectory, a trace, or a transcript. You can grade any slice of it: the final repo, the tools it touched, the arguments it passed, how many steps it burned, whether it deleted .env, latency, tokens, cost.

The principle I keep coming back to: prefer the outcome when the outcome is observable. Two agents can fix the same auth bug through different files and still both be right. If your grader demands one exact tool sequence, you are scoring obedience, not work.

# one task is a record, the suite is a metric

A single trial should leave more than a boolean:

{
  "id": "auth-001",
  "passed": true,
  "score": 1,
  "latencyMs": 8200,
  "inputTokens": 12000,
  "outputTokens": 2300,
  "toolCalls": 7,
  "cost": 0.08
}

Across a few hundred tasks you roll that into suite-level metrics: success rate, pass@1, average score, latency, tokens, cost, tool-call errors, regression rate versus last week.

The score belongs to a task. The metric belongs to the experiment. Mixing them is how a team celebrates “we got 0.91” without knowing whether that is one lucky trial or a hundred honest ones.

# a benchmark is a frozen experiment

Once the dataset and graders stop moving, you can compare systems:

Same tasks. Same isolation. Same checkers. Different agent. Now “v2 is better” is a claim with a denominator.

Dataset ≠ benchmark. An internal set of the last fifty production failures is often more useful than a public leaderboard, and it will never show up on Twitter. OpenAI's Evals repo is useful here as a pattern: a registry of evals, not a single sacred number.

# this is the test suite

The day evals earn their keep is the day a change feels smarter in the demo and worse on the suite.

Agent v1 → 82%
       ↓
change prompt
       ↓
Agent v2 → 76%

After that, agent development starts to look like software:

1. 01

   change the agent — Prompt, model, tools, or policy.

2. 02

   run the evals — Same dataset, isolated trials.

3. 03

   inspect failures — Read the transcript, not just the score.

4. 04

   fix the agent — Or fix the grader, if the grader was wrong.

5. 05

   run them again — The loop is the product.

Anthropic splits this into two moods that I have found useful. Capability evals are supposed to be hard — a hill. Regression evals are supposed to stay near 100% — a fence. When a capability task saturates, it graduates into the fence.

# what the harness actually is

Put the pieces in the order they run:

1. 01Dataset. Load tasks. Each row knows its repo, environment, and how it will be graded.
2. 02Eval runner. For each trial: reset state, start the agent, capture the trace, call graders, write a result row.
3. 03Agent. The same harness you ship — not a toy loop that “should be close enough.”
4. 04Trajectory. Persist tool calls and artifacts. You will read these more than you will read the CSV.
5. 05Graders. Tests, judges, static checks. A task can have more than one. Combine them with a rule you can explain.
6. 06Report. Suite metrics, diffs versus last run, the ten traces you should actually open.

Frameworks — Harbor, Braintrust, LangSmith, a 80-line script — only change how much of that you write yourself. They do not replace the tasks.

# the eval can be the bug

A bad grader can make a good agent look broken, or a broken agent look finished. That is not a footnote. It is the main reliability problem in this whole stack.

Infrastructure is part of the eval, not the backdrop. Anthropic's later note on infrastructure noise is the unglamorous version: extra RAM and CPU moved Terminal-Bench scores by more than the gap between neighboring leaderboard rows. If you do not pin the machine, you are grading the cluster.

# one trial is a sample

Agents are stochastic. The same task can do this:

Trial 1 → PASS
Trial 2 → FAIL
Trial 3 → PASS
Trial 4 → PASS
Trial 5 → FAIL

A single green check is not a rate. If you need “it can find a solution,” people report pass@k — at least one success in k tries. If you need “it works every time a user hits it,” you care about the opposite: how often all k trials succeed. I do not reach for the formula until I know which of those two products I am building.

# the feedback loop is the product

The first two notes in this series were about building the worker: the harness on the laptop, then the computer in the cloud. This one is about the only way those workers stay honest.

BUILD
  ↓
AGENT
  ↓
EVAL
  ↓
FAILURES
  ↓
IMPROVE
  ↓
EVAL
  ↓
REPEAT

Building an agent gives you a system. Evals give you a feedback loop. Without the loop you can still ship a demo that makes people lean in. With it, you can change the prompt on a Tuesday and know by Wednesday what you broke.

That is the whole thesis, still sitting in four boxes:

Dataset
   ↓
Run agent
   ↓
Grader
   ↓
Score

# primary sources

1. 01Anthropic — Demystifying evals for AI agentsTasks, trials, graders, transcripts, capability vs regression, and why graders themselves fail.
2. 02Anthropic — Quantifying infrastructure noise in agentic coding evalsResource limits as a first-class experimental variable on Terminal-Bench and SWE-bench.
3. 03SWE-benchIssue + repo + tests. Verified, multilingual, multimodal, and newer suites live here.
4. 04SWE-bench VerifiedHuman-validated 500-task subset; the default comparison set for a long stretch of 2025–2026.
5. 05SWE-bench GitHubThe evaluation harness and dataset construction.
6. 06Terminal-BenchContainerized, end-to-end terminal tasks — a useful complement once patch-and-test suites saturate.
7. 07OpenAI EvalsA registry-shaped approach to packing tasks and graders.