Home › News
AI Agents · Evaluation · Open Source · Reliability · November 16, 2026

AWS open-sources UAEF: 36 metrics to evaluate AI agents across frameworks

AWS releases UAEF, an open-source agent evaluation system using 36 metrics in seven dimensions across frameworks. We examine regression testing and its limits.

Editorial: AgentenCode · November 16, 2026 · Source-checked analysis
Original editorial composite of a digital AI-agent testing environment with analytical displays, not an AWS product screenshot
Original editorial illustration; not an official product screenshot or interface.

Did the latest agent upgrade actually improve performance?

On November 11, 2026, AWS introduced the Universal Agent Evaluation Framework (UAEF). The open-source system targets a growing problem: a fluent answer does not reveal whether an agent selected the correct tools, avoided unwanted actions or used resources efficiently. A convincing demonstration is not proof of reliable operation.

UAEF aims to evaluate a complete run rather than just the final text. Developers can repeat test cases, persist scores and compare versions after changing a prompt, model or tool. The goal is to make regressions visible before a problematic configuration is rolled out.

Normalizing agent traces across frameworks

LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore produce different execution records. UAEF uses adapters to translate messages, tool calls and handoffs into a common AgentTrace format. AWS also documents support for Langfuse and generic JSON inputs. The evaluator works on that canonical representation regardless of which framework generated the data.

This separation makes evaluation more portable across SDKs and runtimes. A team can compare tool-use behavior across agent implementations without rewriting every metric. However, the quality of the result still depends on complete, representative trace data and suitable expected outcomes for the test case.

Seven dimensions, 36 built-in metrics

AWS describes 36 metrics across Tool Calling, Response Quality, Responsible AI, Performance, Multi-Turn, Multi-Agent and Reasoning. Each measure returns a normalized score from 0 to 1. Separate dimensions can reveal a trade-off: a new model may answer faster while choosing the wrong tool more often.

Some metrics require ground-truth answers or expected tool sequences. Others operate directly on available traces. A single aggregate score can hide a critical failure in an irreversible workflow, so organizations should select metrics for the specific agent task rather than treat the output as a universal intelligence rating.

Diagram: AWS UAEF evaluation and regression testing
Original AgentenCode explainer, not a vendor diagram.

Baselines and regressions are the central opportunity

UAEF supports single-run and dataset evaluation, either by scoring an existing trace or invoking the agent. Persisted experiments use Amazon DynamoDB for metadata and Amazon S3 for detailed results. Teams can compare saved runs against a baseline and review reports explaining where behavior changed.

Imagine a support assistant upgraded to a different model. The final responses still read well, but tool selection for an account-update operation gets worse. A targeted regression dataset can expose that error. The usefulness of the result depends on whether the dataset includes realistic exceptions and high-risk cases, not simply simple success scenarios.

Deterministic measurements versus model-based judges

Certain behaviors, such as whether a specific tool was called with expected arguments, can be tested deterministically against reference data. Other judgments, including contextual relevance, can rely on LLM-as-a-Judge scoring. UAEF supports both families and can integrate other evaluator ecosystems.

Because an agent's output may include malicious instructions, AWS documents protections for its model-based judge: separated rubrics, bounded input and evidence spans that must match the scored text. These measures reduce risks, but a model judgment is still an evaluation signal, not a substitute for real-time security enforcement.

Privacy and operating costs remain outside the headline

Agent traces can contain email addresses, telephone numbers and customer data. UAEF provides PII detection and handling before recorded runs are persisted. This is useful, but it does not prove that every sensitive field is automatically removed or that a deployment meets a specific jurisdiction's privacy rules.

The Python package may be used without provisioning a cloud service. Saving experiments, however, uses S3 and DynamoDB, while LLM-based judges can incur inference costs. AWS also notes that some retained storage and identity resources may survive removal of a deployment stack and continue generating costs unless separately deleted.

Open source improves inspectability, not the reliability of a given agent

The official awslabs/uaef repository provides Apache 2.0 licensed source code, Python APIs, adapters, example notebooks and infrastructure for a web interface. AWS says twelve organizations in telecom, healthcare and finance have used UAEF. That is a vendor-reported adoption claim, not independently verified evidence of a particular productivity gain.

The 36 UAEF metrics are not the same as the 36 AgentenTrust control fields. UAEF measures selected behaviors on test runs; AgentenTrust examines source-documented product controls and assurance claims. The frameworks may complement one another, but neither should be represented as certifying what the other actually measures.

AgentenCode assessment: repeatable quality beats impressive demos

The significance of UAEF lies in a practical constraint: strong language models do not automatically create reliable business workflows. Tool accuracy, unintended side effects, reproducibility and operating costs all matter. Measuring version changes against stable cases is a useful engineering practice.

We would prioritize task-specific acceptance tests over a single benchmark score. Count failed transactions, prohibited tool attempts and human corrections rather than just completed-looking conversations. The Agent Architect can help design the test workflow, while our AgentenTrust overview covers complementary evidence and control reviews. AgentenCode has not independently benchmarked or security-audited UAEF.

Frequently asked questions

What is the Universal Agent Evaluation Framework?

An AWS open-source system for scoring agent execution traces using shared metrics across supported frameworks.

Which agent frameworks are supported?

AWS names LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore, Langfuse and generic JSON adapters.

Do 36 metrics certify an agent as secure?

No. They evaluate defined test behaviors. Runtime access controls and independent security tests remain necessary.

Is UAEF free?

The Apache 2.0 source is open; model inference, stored experiments and cloud infrastructure can still cost money.

RESEARCH & CONTEXT

Sources for this article

This analysis relies on the original publications linked below, which readers can inspect directly.

  1. AWS Machine Learning Blog — UAEF: A framework-agnostic system for closed-loop evaluation of AI agents (11 November 2026)
  2. AWS Labs — Universal Agent Evaluation Framework, source and Apache 2.0 license
  3. AWS Labs — UAEF technical guide and built-in metric catalog

Checked November 16, 2026. Vendor statements, pilot scope and preview benchmarks have not been independently reproduced or certified.

NEWSLETTER · FREE · GERMAN EDITION

More signal. Less AI noise.

Our source-backed German-language digest of major AI developments, agent updates and knowledge articles. The weekly edition is being prepared.

Double opt-in: Subscription starts only after email confirmation. Your details are sent to Brevo when you submit.