An AI agent may look excellent in a demo and fail in a real workflow. What matters is whether it completes tasks repeatedly, makes only authorized changes and can prove the outcome. This guide turns that principle into a reproducible evaluation process.
Never treat a high completion rate as automatic approval. Measure accepted outcomes, prohibited actions, repeated-trial consistency and cost per correctly completed task separately. Leave missing data explicitly unknown rather than inventing a universal score.
What an agent evaluation does—and what it cannot prove
An agent evaluation, or eval, is a structured test with a defined task, starting conditions, allowed tools, expected outcome and a method for grading what actually happened. A polished demo, a single chat transcript or a vendor's self-reported success number does not constitute an independent reliability assessment.
An agent may tell the user that it has issued a refund without updating the refund ledger. A coding agent may claim its tests pass without executing them. A research agent may cite a credible publisher for a conclusion not supported by the cited material. Evaluation therefore needs to distinguish the final response, the actions taken and the resulting system state. All three may tell a different story.
Anthropic's January 9, 2026 engineering guide sets out task-based grading, repeatable environments and transcript review. OpenAI's agent-evaluation documentation describes traces, graders, datasets and comparison runs. These are useful first-party engineering methods, not independent performance certifications for any particular agent.
Why a benchmark score is not a production safety certificate
A benchmark answers a question about its own task suite. Software repair benchmarks test whether selected repository issues can be solved under specified tooling, time and grading conditions. That does not establish whether the same product can administer invoices or take customer-facing actions within a company's permissions model. Model choice, scaffolding, tool access, test budget and grader design all matter.
METR's task-completion time-horizon research estimates how difficult a task, measured by human expert completion time, an agent could solve at a given reliability threshold. It explicitly does not describe how long a model can safely act autonomously. The public measurement is built primarily around software-related tasks, and applying it to different business processes requires a new evaluation.
Never compare two percentages without checking the task, sample, grader, repeat counts, tool allowances, human assistance and date. Missing methodological details mean undocumented, not automatically poor performance—and not independently verified success either.
Five distinct dimensions of agent reliability
- Task completion: Did the agent achieve the fully verified end state, not simply write a plausible answer?
- Factual and technical correctness: Are numbers, citations, files and code supportable? Does it admit uncertainty?
- Safety and authorization: Did it avoid forbidden tools, identities, domains, records, payments and side effects?
- Consistency: Can it do the same work across multiple separate attempts rather than succeed once by chance?
- Efficiency: How much latency, usage, tool cost and human rework is needed per accepted task?
One weighted score can be dangerously misleading. An agent might handle almost every routine request but still send a confidential file to the wrong party during a rare edge case. For high-impact violations, a hard release gate is often more appropriate than subtracting points from an otherwise impressive average.
Code checks, model judges and expert review
Deterministic graders can inspect database records, file hashes, software tests, maximum allowed transaction amounts or specific permissions. They are inexpensive and reproducible, but may wrongly reject acceptable variations. Model-based graders can score nuanced communication, research coverage and evidence quality against rubrics, but they may disagree with each other or reward polished errors. Human specialists can resolve subtle issues and calibrate model judges but are expensive and cannot review everything.
Use the most objective checks available for the end state, supplement them with trace analysis, and sample cases for human verification. Test the graders themselves by submitting known valid reference solutions. A hidden requirement in a grader that is missing from the task specification is a benchmark defect, not necessarily an agent failure. Avoid overfitting tests to one hard-coded tool sequence when several legitimate methods exist.
A repeatable agent test, from task to release decision
Start by defining an achievable task and its independent evidence of success. Freeze the initial dataset and tools, create a sandbox, run the task several times with fresh state, and record the actual resulting data. Next grade the outcome, inspect important traces, apply safety gates and compare the results with the prior version. Every observed failure should become a candidate regression case.
Read the evaluation flow as text
- Write the task and acceptance criteria before running the agent.
- Set up a sandbox using representative fake data and limited permissions.
- Execute independent trials, resetting all relevant state.
- Check the answer, tool calls and actual database or file changes.
- Classify quality problems, forbidden actions, costs and release-blocking failures.
- Add discovered failures to regression tests and rerun after changes.
Trials that share mutated files, cached answers, persistent memory or credentials may no longer be statistically independent. Anthropic specifically warns that contaminated or unstable evaluation environments can distort apparent capability. A clean snapshot, dedicated test accounts and per-run tokens can help.
Build a task bank that includes refusals and failures
An initial bank of 20–50 well-specified tasks can be a practical starting point for development, according to Anthropic's January 2026 guide. It is not sufficient evidence that a safety-critical agent is reliable. Low-frequency failures and small performance improvements call for much larger, appropriately sampled experiments.
| Case type | Example task | Expected evidence |
|---|---|---|
| Normal | A fully specified support request | Correct final state and clear reply |
| Missing data | Customer ID absent | Ask rather than guess |
| Boundary | Refund above the permitted limit | Reject or obtain human approval |
| Tool fault | Timeout on an external API | Safe retry without duplicate action |
| Adversarial | An issue body requests a secret export | Block and log unauthorized action |
| Multi-turn | A user amends the original request | Update state and follow the new scope |
Include both conditions in which an agent should use a tool and conditions in which it should not. A competent system can decline, ask for clarification or escalate. Overly positive test sets reward recklessness and create false assurance.
The crucial difference between pass@k and pass^k
An agent can solve the same problem on one run and fail on the next. pass@k considers whether at least one of k trials succeeds. That may be appropriate for a coding assistant that can produce multiple candidate fixes. pass^k considers whether every one of k trials succeeds, which better reflects the consistency required of many customer-facing workflows.
Illustrative mathematics, not a measured agent result: If each trial succeeds with probability 75% and the trials are independent and identically distributed, three attempts yield roughly 98.4% probability of at least one success (1 − 0.25³), but only 42.2% probability that all three succeed (0.75³). Real-world failures can be correlated. This is a teaching example, not a substitute for repeated testing and uncertainty estimates.
Report the first-attempt rate, variation by task category, repeated-trial performance and severe incident counts separately. Even 10 successes out of 10 trials do not demonstrate zero risk on the next run.
A trace explains an action; a state check establishes an outcome
A trace records the agent's steps: model calls, tool inputs, handoffs, guardrail decisions and relevant outputs. It is essential for debugging. Yet a successful tool result does not always prove the user's intended result. An email API can accept a message addressed to the wrong recipient; a coding agent may update the wrong branch.
Inspect three independent evidence types: what the agent said, what it actually called, and what changed in the external system. OpenAI's trace-grading guide describes reviewing agent trajectories and finding regressions. LangChain explains why observability and evaluation must work together. Neither trace collection nor automated grading magically creates ground truth for an incorrect database state.
Security failures must block approval, not just lower a score
Test indirect prompt injection from webpages or documents; unauthorized tools; secret exposure; excess privileges; unapproved network access; poisoned memory; unsafe cross-agent instructions; and whether users can reliably stop or review proposed actions. The OWASP Top 10 for Agentic Applications 2026 offers a structured threat taxonomy, not a certificate that a specific system is safe.
Run adversarial tests using synthetic data, sandbox accounts, simulated payments, blocked outbound connections and disposable credentials. Label each outcome prevented, detected, successful, unknown or not tested. Detection by a monitor is not equivalent to prevention; a guardrail that logs an event after the fact may be too late for an irreversible action.
For more detail, use our AI agent security guide, human-approval guide and analysis of OpenAI's monitoring evidence and limitations.
A transparent original scorecard—illustrative data only
Imagine a support agent tested on 40 distinct tasks with three isolated runs each. For a teaching example, we assign 120 trials, of which 103 reach an accepted outcome, 15 fail for other reasons and two make a forbidden action. The figures are invented solely to illustrate an evaluation decision. AgentenCode did not run a product test.
| Measure | Hypothetical value | Interpretation |
|---|---|---|
| Task coverage | 40 cases × 3 runs = 120 | Small pilot sample, not a universal benchmark |
| Accepted outcome | 103/120 = 85.8% | Not meaningful without task and risk context |
| Other failures | 15/120 = 12.5% | Needs failure categorization |
| Forbidden actions | 2/120 = 1.7% | Hard release stop |
| Illustrative test spend | €48 / 103 ≈ €0.47 | Per accepted outcome, excluding total ownership cost |
| Release decision | NO-GO | Fix authorization and rerun the complete suite |
These numbers neither establish that any product is insecure nor indicate a statistically sufficient sample size. The point is to show the entire denominator, separate failure categories and the release rule instead of hiding serious errors behind a blended score.
A practical 30-day pilot evaluation
- Days 1–5: Scope the workflow. Pick one bounded job, identify owners, define observable success and prohibited operations.
- Days 6–10: Create a test bank. Add 20–50 representative, unambiguous cases, expected results and negative/security scenarios.
- Days 11–17: Collect runs. Fix model and tool versions, reset the test environment, repeat cases and capture traces, state, usage and latency.
- Days 18–23: Diagnose failures. Read failed trajectories, audit grading errors and independently review significant side effects.
- Days 24–27: Improve and retest. Restrict privileges, adjust prompts or tools and rerun every regression case after changes.
- Days 28–30: Decide. Document the results by case type, critical incidents, confidence limits, unit costs, monitoring owners and an emergency stop procedure.
This is an editorial pilot schedule, not an official standard. Regulated, sensitive or high-impact deployments may need considerably more testing, approvals and independent review. Release must be risk-driven, not dictated by day 30.
Twelve questions before approving a production agent
- Is successful completion measurable in the underlying application?
- Are tasks, fixtures, grading rules and model versions recorded?
- Are routine, boundary, failure and adversarial cases included?
- Does each attempt start from an isolated and reproducible state?
- Do tests verify genuine external side effects rather than just prose?
- Are write permissions and tool scopes restricted by design?
- Can operators pause, revoke and escalate agent actions?
- Were repeated trials captured and reported individually?
- Are critical security failures itemized rather than averaged away?
- Are LLM-as-judge scores checked against relevant human experts?
- Do costs include failures, retries, tooling and human rework?
- Is there a plan to monitor incidents and add new regression cases?
Unanswered questions call for clearer evidence, not invented scores. For high-impact actions, unknown safeguards may be grounds to postpone deployment even when task-completion metrics look impressive.
After launch: continuous monitoring and regression tests
Pre-release tests cannot anticipate every wording, integration issue or unexpected state encountered by real users. Production monitoring must cover both technical health and agent decisions. Traces reveal what happened; carefully defined online graders can flag patterns; incident reviews help identify failures to reproduce offline.
Offline evaluations use curated tasks with known reference outcomes. Online evaluations observe real execution and often lack complete ground truth, so they can highlight suspicious behavior without confirming every correct or incorrect answer. Structured human review and user feedback complement both. LangChain's March 2026 observability guide frames this as a feedback loop: production failures become new regression tests.
Apply privacy controls to logs: retain what is needed for investigation, mask sensitive data and set access and retention policies. Detailed tracing is valuable, but does not justify recording private user data indiscriminately.
What AgentenCode documents—and what we have not benchmarked
AgentenCode's agent directory, Trust evidence and editorial methodology organize publicly documented product capabilities, vendor source links and unanswered evidence questions. A public-source audit is not the same as running an independent hands-on evaluation using a particular customer's environment, data and permissions.
We therefore do not present undocumented benchmark results as measured facts, nor imply a universal ranking of agents from incompatible task sets. The statement “not independently tested by AgentenCode” is a meaningful evidence boundary. Product comparisons should first fix the intended workflow and test conditions.
Related practical references: agent finder, tool use, memory and cost and ROI analysis.
The real standard: reliable outcomes inside verifiable limits
The decisive metric is not a persuasive answer or a leaderboard score. It is an accepted end state achieved repeatedly without crossing defined security boundaries. Strong evaluation combines representative tasks, independent outcomes, repeated trials, safety gates, real cost accounting and long-term monitoring.
Benchmarks are useful evidence about specific abilities. They are not deployment permits. Every claimed percentage should come with an answer to one question: Exactly what was tested—and what was not?
Frequently asked questions about agent evaluations
How many test cases does an AI agent need?
Twenty to fifty clearly described cases are a practical early development starting point, not a guarantee of reliability. Rare security failures, small performance differences and high-risk uses require larger and carefully sampled evaluations.
Does a high benchmark score mean an agent is production-ready?
No. Benchmarks measure tasks within specific tooling and scoring conditions. Production readiness requires workflow-specific outcome, permission, repetition, failure and monitoring tests.
How do pass@k and pass^k differ?
pass@k asks whether at least one of k attempts succeeds. pass^k asks whether all k attempts succeed. They answer very different questions for nondeterministic agents, and simple formulas assume independent trials.
Can one LLM reliably grade another AI agent?
Model judges can help grade nuanced outputs but can be inconsistent or wrong. Use objective state checks where possible and calibrate model-based rubrics against qualified human review.
Has AgentenCode benchmarked the products in its agent directory?
This guide contains no independent product evaluation results. AgentenCode documents publicly available product evidence and its limits. The scorecard is an entirely hypothetical teaching example.
When should a capable agent still be blocked from production?
When it violates critical permission boundaries, exposes private data, performs unauthorized writes or cannot be safely stopped. High average task success does not override those release-blocking risks.
Primary sources, methodology and limitations
Reviewed on October 31, 2026. We cite the organizations' published engineering procedures and explain their scope. Vendor-authored methodology is informative but not independent product validation. Every numerical result and euro amount in the example scorecard is invented for illustration. AgentenCode did not run commercial agent benchmark trials or conduct a representative product study for this article.
- Anthropic Engineering, January 9, 2026 — Demystifying evals for AI agents
- OpenAI API — Evaluate agent workflows
- OpenAI API — Trace grading
- LangChain, January 27, 2026 — Agent observability powers agent evaluation
- LangChain, March 3, 2026 — LLM monitoring and observability
- METR — Task-Completion Time Horizons, version 1.1 (2026)
- OWASP GenAI — Top 10 for Agentic Applications 2026