How to evaluate AI agents: tasks, graders, pass@k and pass^k
How to evaluate AI agents: build tasks from real failures, grade outcomes, run several trials and report pass@k and pass^k. With a runnable Python demo.

To evaluate an AI agent, you give it a set of realistic tasks in a clean, isolated environment, run each task several times, and grade the final state of the environment (did the booking exist, did the tests pass) with code where possible and an LLM or human grader where not. Then you report how often it succeeds at least once (pass@k) and how often it succeeds every time (pass^k), and you read the transcripts before you believe any of the numbers.
Why are agents harder to evaluate than a single LLM call?
A single prompt has one input and one output. An agent takes many turns, calls tools, changes state and adapts as it goes, so a mistake in step three can surface in step twelve. Anthropic’s Demystifying evals for AI agents (January 9, 2026) is the best practical guide I’ve read on this, and much of this post follows its structure.
It also gives a useful vocabulary:
- A task is one test with defined inputs and success criteria.
- A trial is one attempt at a task. You run several, because outputs vary between runs.
- A grader scores some aspect of the result; a task can have several.
- A transcript (trace, trajectory) is the full record of a trial: messages, tool calls, results.
- The outcome is the final state of the environment. An agent that says “your flight has been booked” hasn’t booked anything; the outcome is whether a reservation exists in the database.
- The evaluation harness runs it all. The agent harness is the thing being tested, which means you’re never evaluating a model alone, you’re evaluating a model inside its agent harness.
What should agent evals measure?
Two kinds of suites, with different targets.
Capability evals ask what the agent can do. They should start with a low pass rate, giving you a hill to climb.
Regression evals ask whether it still does what it used to. They should pass close to 100% of the time, and any drop means something broke. Anthropic’s advice is that capability tasks the agent has mastered “graduate” into the regression suite.
Within a task, grade the outcome first. For a coding agent, that’s whether the tests pass, which is exactly how SWE-bench works. For a support agent, it’s the database state at the end of the conversation, which is how τ-bench grades. Then add transcript checks where they matter: turn count, forbidden tool calls, tone.
How do you build an agent eval suite?
Anthropic’s roadmap, condensed, with my notes:
- Start early, and small. “20-50 simple tasks drawn from real failures is a great start.” Early changes have large effects, so small samples are enough to see them.
- Start from what you already check by hand. The manual checks before each release, your bug tracker, your support queue.
- Write unambiguous tasks with reference solutions. The test: two domain experts would independently reach the same pass or fail verdict. With frontier models, a 0% pass rate across many trials usually means the task is broken, not the agent.
- Balance the set. Test when a behavior should happen and when it shouldn’t. If you only test that the agent searches when it should, you get an agent that searches for everything.
- Isolate every trial. Start each from a clean environment. Anthropic saw Claude gain an unfair advantage on some tasks by reading the git history left from earlier trials.
- Grade the outcome, not the path. Checking for an exact sequence of tool calls is brittle, because agents find valid routes you didn’t anticipate. Give partial credit for multi-step tasks.
- Read the transcripts. This is the step people skip and the one that tells you whether a failure is the agent’s or the grader’s.
- Watch for saturation. A suite at 100% only catches regressions. Add harder tasks.
How do pass@k and pass^k differ?
Both come from running each task n times and counting c successes.
pass@k is the probability that at least one of k attempts succeeds. It comes from code generation, where you can run tests on several candidates and keep one that passes; the HumanEval paper defined an unbiased estimator for it. It goes up as k grows.
pass^k (“pass hat k”), proposed in τ-bench, is the probability that all k attempts succeed. It measures consistency, and it goes down as k grows. Anthropic’s example: an agent with a 75% per-trial success rate passes all three of three trials only (0.75)³, about 42%, of the time. τ-bench found that even state-of-the-art function-calling agents such as gpt-4o succeeded on under 50% of tasks, and pass^8 in the retail domain was below 25%.
Which one you report is a product decision. A coding agent whose candidates are checked by tests can use pass@k. A customer-facing agent that must behave the same way every time should be judged on pass^k, because users meet every trial, not the best one.
What do the numbers look like in practice?
This script simulates a 30-task suite with eight trials per task, computes both metrics with their unbiased estimators, and puts an error bar on a single run’s score.
# agent_metrics.py: pass@k, pass^k and an honest error bar for a small agent eval.
# Python 3 standard library only. Trial outcomes are simulated with a fixed seed.
import math, random
random.seed(7)
N_TASKS, N_TRIALS = 30, 8
true_rate = [random.choice([0.95, 0.75, 0.5, 0.2]) for _ in range(N_TASKS)]
trials = [[random.random() < p for _ in range(N_TRIALS)] for p in true_rate]
def pass_at_k(n, c, k): # unbiased estimator from Chen et al. (2021)
return 1.0 if n - c < k else 1 - math.comb(n - c, k) / math.comb(n, k)
def pass_hat_k(n, c, k): # all k of k trials succeed (tau-bench's pass^k)
return math.comb(c, k) / math.comb(n, k)
for k in (1, 2, 4, 8):
at = sum(pass_at_k(N_TRIALS, sum(t), k) for t in trials) / N_TASKS
hat = sum(pass_hat_k(N_TRIALS, sum(t), k) for t in trials) / N_TASKS
print(f"k={k} pass@k={at:5.1%} pass^k={hat:5.1%}")
def wilson(successes, n, z=1.96): # 95% interval for a proportion
p = successes / n
centre = (p + z*z / (2*n)) / (1 + z*z / n)
half = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / (1 + z*z / n)
return centre - half, centre + half
first = sum(t[0] for t in trials)
lo, hi = wilson(first, N_TASKS)
print(f"\nsingle run: {first}/{N_TASKS} solved = {first/N_TASKS:.0%}, 95% interval {lo:.0%} to {hi:.0%}")
print(f"three trials at 75% each all pass: {0.75**3:.0%}")
Output (Python 3.13.11):
k=1 pass@k=69.6% pass^k=69.6%
k=2 pass@k=81.2% pass^k=58.0%
k=4 pass@k=89.7% pass^k=47.0%
k=8 pass@k=96.7% pass^k=36.7%
single run: 20/30 solved = 67%, 95% interval 49% to 81%
three trials at 75% each all pass: 42%
At k=1 the two metrics are the same number. By k=8 they are 60 points apart: the same agent “almost always finds a solution” and “is reliable on about a third of tasks”. Both statements are true, and a demo only ever shows you the first.
The last two lines are the part teams skip. Thirty tasks run once gives 67%, with a 95% interval from 49% to 81%. A prompt change that moves the score from 67% to 73% on that suite tells you nothing. Run more trials, add tasks, or compare on the same tasks with a paired test before you call it a win.
Which graders should you use?
Anthropic splits graders into three types, and the order of preference is clear: “deterministic graders where possible, LLM graders where necessary or for additional flexibility, and using human graders judiciously.”
- Code-based: tests, state checks, exact match, static analysis. Fast, cheap, hard to argue with.
- Model-based: an LLM as a judge with a rubric, for tone, completeness or whether a research summary is grounded. Calibrate it against human labels, give it an “Unknown” option, and grade one dimension per judge call.
- Human: for calibration and for subjective outputs where you need the ground truth.
Graders have bugs too. Anthropic’s post describes Opus 4.5 scoring 42% on CORE-Bench until a researcher found rigid grading (it rejected “96.12” when expecting “96.124991…”), ambiguous specs and irreproducible tasks; after fixes it scored 95%.
Where do agent evals go wrong?
Measuring the model, forgetting the harness. Change the tools, the system prompt or the context strategy and the score moves. Version all of it together.
One trial per task. Averages over single runs hide inconsistency. That’s the pass^k lesson.
Shared state between trials. Leftover files and caches cause correlated failures, or correlated wins.
Grading the path. Agents find better routes than the ones you scripted. Anthropic describes Opus 4.5 “failing” a τ2-bench flight-booking task by finding a loophole in the policy that served the user better.
Trusting public benchmarks for your product. Public benchmarks leak into training data; OpenAI dropped SWE-bench Verified for that reason. Your own tasks don’t have that problem. More on that in what LLM benchmarks measure.
What I would do
- Write 30 tasks from real failures this week. Bug reports, support tickets, the cases you check by hand. Each gets a reference solution and a code-based outcome check.
- Run five trials per task and report pass@1 and pass^5 side by side, plus cost and latency per task.
- Put evals in CI for every prompt, tool or model change, and keep a regression suite that must stay near 100%.
- Read ten transcripts per run. Failures first, then a few passes, to catch passes for the wrong reason.
- Add an LLM judge only where code can’t grade, and check it against a few dozen human labels before trusting it.
- Re-run the suite on every new model. Anthropic’s point that teams with evals can adopt a new model in days, not weeks, matches how I’d want to run delivery: the eval suite is the acceptance test for a model swap, the same way the agentic engineering loop treats tests as the contract.
FAQ
What is the difference between pass@k and pass^k?
pass@k is the chance that at least one of k attempts succeeds; it rises with k. pass^k is the chance that all k attempts succeed; it falls with k. Use pass@k when you can pick the best attempt, pass^k when every attempt reaches a user.
How many test cases do I need for agent evals?
Anthropic suggests starting with 20 to 50 tasks drawn from real failures. That’s enough to see large effects early on. To detect small improvements you need more tasks, more trials, or both; check the confidence interval before trusting a small gain.
Can I use an LLM to grade an AI agent?
Yes, for things code can’t check, like tone or the quality of a summary. Calibrate the judge against human labels first, grade one criterion per call, and prefer outcome checks in code wherever they exist.
What frameworks exist for agent evaluation?
Anthropic’s post lists Harbor (containerized agent runs, used by Terminal-Bench 2.0), Braintrust, LangSmith, Langfuse and Arize Phoenix. A short Python script and a folder of task files is also a fine place to start.
Sources
- Anthropic, Demystifying evals for AI agents, January 9, 2026
- Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024
- Chen et al., Evaluating Large Language Models Trained on Code, July 2021 (pass@k estimator)
- OpenAI, Why we no longer evaluate SWE-bench Verified, February 23, 2026
- Anthropic, Building effective agents, December 19, 2024