Agent Evaluation Guide

TL;DR

• Agent evaluation asks whether the system actually completed the job — not whether a single model response looks fluent. An agent may reason, call tools, retrieve data, wait on humans, delegate, retry, and fail before any final answer exists.

• Score two things: the outcome and the trajectory. A correct summary produced by calling the wrong tool, looping five times, or leaking another customer's order is not a successful run.

• For many agent systems, the execution trace is the primary evaluation artifact. Instrument the loop, then score spans (tool selection, arguments, retrieval), the full trajectory (plan, recovery, efficiency), and the task result (success, groundedness, safety, cost). Component tests and end-state oracles still apply.

• Offline suites catch regressions; production sampling catches reality. Mature teams run both, then feed failures back into a versioned evaluation dataset.

• LLM-as-a-judge is a scorer, not ground truth. Rubrics, calibration against humans, and deterministic checks for tools and schemas keep scores honest.

Why This Matters

The question behind every production agent is simple: how do you know whether it actually works?

A chatbot can be sampled by reading answers. An AI agent is a control loop. It may select tools, call them more than once, retrieve private data, mutate external state, hand work to another agent, or stop before producing text. The failure is often invisible in the final message: the order was updated twice, the refund went to the wrong account, or the agent asked a knowledge base after inventing a tracking number.

Traditional LLM evaluation still matters for the final utterance. It is not sufficient for the loop. You need signals for task completion, tool-call accuracy, trajectory quality, groundedness, policy compliance, and operational cost — the same way you would not ship a workflow engine by reading only its last log line.

This guide is the evaluate layer of the agent stack: build → run → trace → evaluate → improve. Observability records what happened. Evaluation decides whether it was acceptable, then turns failures into regression tests.

Walk a measured loop in the Agent Loop Lab and a tool-decision loop in the Tool Calling Lab before you design scorers — the traces those labs expose are the same objects production eval consumes.

The Problem Agent Evaluation Solves

Without agent evaluation, teams cannot answer:

1. Did the agent finish the user's objective, or only produce a confident paragraph?
2. Did it take a reasonable path — or wander, retry blindly, and get lucky?
3. Which component failed — prompt, model, tool schema, retrieval, memory, permissions, or an upstream API?
4. Is this change a regression? A new model, prompt, or tool often improves demos and quietly breaks edge cases.
5. Is a "successful" agent operationally viable? Latency, token use, and cost per task can make a correct agent unshippable.

Single-turn metrics hide all of this. Exact match on the final string treats a four-tool recovery the same as a one-tool lookup. Pass@1 on a benchmark does not tell you whether the agent called refund_order without approval. User thumbs-down arrives after the damage.

Agent evaluation inserts a measurement layer over traces: score the run, classify the failure, and gate the next change.

How We Got Here

Agent evaluation reused LLM scoring, then had to grow a second axis: actions.

Diagram: Evolution of agent evaluation

flowchart LR
    A[Tool traces] --> B[Task benches]
    B --> C[Trajectory scores]
    C --> D[Offline plus online]

Conceptual progression, not a dated chronology. Output scoring was not enough once models started calling tools.

Early agent papers treated the trace as an explanation (ReAct, 2022). Interactive benchmarks such as AgentBench and GAIA (2023) scored task success in environments; function-calling leaderboards isolated schema fidelity; SWE-bench-style tasks scored end-state correctness. Production practice added offline golden tasks for regression and sampled online evaluation for drift — a loop familiar from classical ML, applied to agent runs. The durable idea is not a vendor: persist the run, score outcome and trajectory, classify failures, re-run after every change.

Open-source agent stacks in the 2026 GitHub ranking and the agents topic show the runtimes to evaluate — loops, tools, handoffs — not chat-model leaderboards.

What Is Agent Evaluation?

Agent evaluation is the systematic measurement of whether an agent run achieved its objective, and whether the sequence of model calls, tool calls, retrievals, and handoffs that produced that result was acceptable under quality, safety, and efficiency constraints.

It is a specialization of AI evaluation, not a replacement. Retrieval, generation, prompts, and RAG still need their own layers (retrieval evaluation, prompt evaluation, RAG evaluation). Agent evaluation adds the control-loop dimensions those layers do not cover: tool selection, argument validity, trajectory shape, recovery, delegation, and task-level success when the "output" is a side effect rather than a sentence.

The central distinction:

• What the agent produced — final answer, updated ticket, refund, patch, or "I cannot do that."
• How the agent got there — which tools, in what order, with which evidence, at what cost, and within which policy bounds.

A run can pass one and fail the other. Treat that as a first-class result, not an annoyance.

Why Evaluating Agents Is Different

Evaluating agents is different from scoring a single LLM response because the system is a stateful, tool-using process in an environment you do not fully control.

• Multi-step execution — a polite final paragraph can hide a skipped lookup.
• Tool use — wrong tool → wrong evidence → fluent hallucination (tool calling).
• External state — success is often a ticket field or DB row, not a string match.
• Non-determinism — several valid paths exist; gold sequences over-penalize harmless variation.
• Long trajectories — loops, early stops, and context bloat are quality and cost signals.
• Partial success — found the order but not the delay reason is not the same as a total miss.
• Retries — one recovered timeout is not six calls to a dead tool.
• Delegation and memory — handoffs and stale session state add failure modes (multi-agent systems, agent memory).
• Environment — clock, inventory, and sandbox vs prod APIs must be declared in the eval fixture.

Example: "Cancel order 12345 if it has not shipped." LLM eval checks the reply. Agent eval checks that get_order ran, status == unshipped was respected, cancel_order ran once, and a shipped order was refused.

What Should You Evaluate?

Not every agent exposes an explicit plan, memory store, or multi-agent graph. Pick dimensions that match the runtime — a menu, not a mandatory scorecard.

Diagram: Agent evaluation dimensions

flowchart TB
    Run[Agent run]
    Run --> Task[Task success]
    Run --> Final[Final answer]
    Run --> Traj[Trajectory]
    Run --> Safe[Safety]
    Run --> Eff[Efficiency]
    Traj --> Tools[Tool calls]
    Traj --> Plan[Planning]
    Final --> Ground[Groundedness]

Outcome, path, evidence, policy, and cost are separate scores on the same run; collapsing them into one number hides the failure.

Task success

Did the agent meet the user's objective? Score success, failure, and partial completion; prefer environment end-state (ticket updated, tests passing, refund issued once) when tools mutate state. Necessary, not sufficient: a completed task can still violate policy or blow the budget.

Final answer quality

Was the user-visible output correct, relevant, complete, grounded, and useful? Apply LLM evaluation here. Alone it is misleading: a perfect explanation of a fabricated tracking event is a high answer score and a failed task.

Trajectory quality

Was the sequence of actions reasonable? Flag loops, redundant tools, missing required steps, and premature termination. Prefer constraints (required/forbidden tools, max steps) over a unique gold path unless the domain truly has one.

Tool-call accuracy

Score these separately: selection (right tool for the state), arguments, execution success, result interpretation, and recovery. A selected tool can still fail; a failed call that recovers is not the same as a blind loop. Often the highest-leverage slice for tool-using agents — see the Tool Calling Lab.

Planning quality

Where a plan is exposed or implied, score decomposition, ordering, dependencies, and replanning. Many ReAct-style agents never emit a plan object; infer planning from the trajectory (agent planning).

Groundedness / factuality

Are claims supported by tool results, retrieval, or other declared sources? Uncited figures after a successful file read are still a groundedness failure. When the agent retrieves, reuse RAG evaluation.

Safety / policy compliance

Score unsafe actions, unauthorized tools, prompt injection (including in tool output), leakage, and permission violations. Record whether guardrails blocked the action or HITL escalated it.

Efficiency

Latency, tokens, model calls, tool calls, duration, and cost per task. A successful agent can still be operationally poor. Report efficiency with quality — it does not replace it.

Agent Evaluation Metrics

Use metrics that map to a failure you can fix. Not every metric applies to every agent. A read-only research agent has no human-escalation rate; a workflow-shaped agent may not need planning scores.

Report quality and efficiency together. A 4-point gain in task success that triples cost is a product decision, not an automatic win.

Evaluation Levels

These levels are a practical hierarchy, not a standardized industry taxonomy. They complement each other. A green task score with a red tool-selection score is a debugging clue, not a contradiction.

Diagram: Evaluation levels

flowchart LR
    L1[L1 Components] --- L2[L2 Steps]
    L2 --- L3[L3 Trajectory]
    L3 --- L4[L4 Task]
    L4 --- L5[L5 Production]

Complementary granularities, not a required waterfall. Lower levels locate the bug; higher levels decide whether the system is fit for users.

Level 1 catches a broken tool stub before you burn a golden set. Level 5 catches the prompt injection that only appears in live ticket text. Skipping levels produces "the score dropped" with no owner.

How Agent Evaluation Works

The mechanism is the same regardless of framework:

1. Define the task distribution and success criteria (not just "be helpful").
2. Instrument the agent so each run emits a trace: model calls, tool calls, retrieval, handoffs, final output, errors.
3. Apply scorers to spans, to the trajectory, and to the outcome. Mix deterministic checks with judges and humans.
4. Aggregate by slice: task type, tool, model version, prompt version, tenant.
5. Gate changes on offline regression; sample production to refresh the dataset.

Observability is the collection plane. Evaluation is the judgment plane. Component tests and end-state oracles work without a full trace; attributing which step failed usually requires reconstructing the run.

Offline vs Online Evaluation

Mature systems use both. Offline eval is the regression suite. Online eval is sampled judgment of the distribution you actually serve — not a requirement to score every request in real time.

Offline without online overfits the suite. Online without offline means you learn about regressions from users.

Decision tree: Offline vs online vs human review

flowchart TD
    Off[Offline suite] --> On[Sample production]
    On --> Hum{High impact?}
    Hum -->|Yes| HR[Human review]
    Hum -->|No| Auto[Auto scores]

Regress offline on fixtures and replayed runs; sample production for the live mix; spend humans on ambiguous, unsafe, or high-impact slices. Online eval is sampled judgment, not scoring every request in real time.

How to Build an Agent Evaluation Dataset

An evaluation case should be executable, not a vibe check. A useful case typically includes:

• Input / task — the user request and relevant session context
• Environment — available tools, data fixtures, time, permissions
• Expected outcome — end state and/or reference answer
• Success criteria — what "done" means, including partial credit
• Optional expected actions — required tools, forbidden tools, not a full gold path unless necessary
• Metadata — severity, failure mode, source (prod, synthetic, adversarial)

Sources that actually improve systems:

• Real production tasks (redacted) — the prior you care about
• Historical failures — tickets, thumbs-down, HITL rejects, incidents
• Manually authored scenarios — rare but expensive paths (refunds, deletions)
• Synthetic tasks — fill coverage holes once the schema of a case is stable
• Adversarial scenarios — injection in user text and in tool payloads
• Public benchmarks — useful for research comparison; insufficient as a product gate

Emphasize failure modes you have seen, not only happy paths. A 100-case suite of clean lookups will not catch the agent that cancels shipped orders.

Version the dataset in git (or an eval store with versions). Pin it in CI with model ID, prompt version, tool schema version, and retriever version. When production fails, add a case before you "fix the prompt."

Trace-Based Evaluation

Modern agent evaluation often operates on execution traces, not on a lone completion string. For many tool-using systems that is the primary artifact; it is not the only one.

Diagram: Trace-based evaluation

sequenceDiagram
    participant U as User
    participant A as Agent
    participant M as Model
    participant T as Tools
    U->>A: Task
    A->>M: Step
    M-->>A: Tool intent
    A->>T: Tool call
    T-->>A: Result
    A->>M: Observation
    M-->>A: Final
    A-->>U: Answer

The application executes tools; the model proposes intents. Evaluation can attach scores to spans, to the whole trajectory, and to the outcome.

Conceptually:

User task → agent → model call → tool call → tool result → model call → tool call → final answer.

Evaluate at three granularities:

• Span / step — this tool name, these arguments, this retrieval hit list, this guardrail verdict
• Trajectory — order, loops, missing required steps, recovery, handoffs
• Final outcome — task success, answer quality, safety, cost, latency

Vendor-neutral tracing (OpenTelemetry-style spans, or your own run_id + event log) is enough. The Agent Loop Lab shows the same structure: decide, act, observe, terminate. Production eval is that log plus scorers.

Store enough to replay: tool schemas, arguments, truncated results, model and prompt versions. Redact PII before the trace leaves the trust boundary — evaluation does not require raw account numbers if stable IDs and fixtures exist.

LLM-as-a-Judge

LLM judges scale rubric scoring when exact match cannot: "was the delay explanation faithful to the tracker payload?" They are useful for trajectory critiques and pairwise comparisons of two agent versions.

They are not ground truth.

Where judges work reasonably well:

• Faithfulness to provided tool/retrieval evidence
• Rubric dimensions with anchored scores (0 / 0.5 / 1.0)
• Pairwise "which trajectory wasted fewer steps?" with order swaps

Where they fail:

• Precise tool-argument correctness (use schema and equality checks)
• Hidden side effects the judge cannot see
• Safety edge cases the judge model shares with the agent model
• Long traces that exceed the judge context, causing missed loops

Practices that keep judges from becoming a second untested agent:

• Rubric-based scoring with explicit anchors, not "rate 1–5 helpful"
• Pairwise comparison for A/B on prompts/models, with swapped order
• Reference-based judging when a gold answer or gold facts exist; reference-free when you score against a rubric or evidence in the trace without a gold answer
• Calibration against human labels; track an appropriate inter-rater statistic (for example Cohen's κ on suitable categorical labels) and retire judges that drift
• Separate judge family from the agent model when possible, to reduce self-preference
• Deterministic pre-checks so the judge never scores malformed JSON as "mostly correct"

Known issues: position bias, verbosity bias, model preference, prompt sensitivity, and correlated errors (agent and judge both trust a hallucinated tracker field). If those show up, the metric is measuring judge taste, not agent quality.

Human Evaluation

Humans remain necessary when the task is ambiguous, quality is subjective, safety is at stake, the decision is high-impact, or the workflow is too complex for a stable rubric.

Use humans to:

• Define and revise rubrics
• Label a calibration set for judges
• Review sampled production traces (especially HITL rejects and safety flags)
• Adjudicate disagreements

Process matters as much as taste: written rubrics, double annotation on a slice, inter-rater agreement, and reviewer calibration sessions. Uncalibrated reviewers produce scores you cannot regress on.

This is adjacent to human-in-the-loop but not the same thing. HITL is a runtime control (approve the refund). Human evaluation is a measurement control (score whether that refund should have been proposed). Many teams reuse the same reviewers for both; keep the artifacts distinct so ops queues do not silently become your only eval set.

Evaluating Tool-Using Agents

Tool-using agents should be scored as an API client with a language model attached.

Evaluate, in order:

1. Tool selection — right capability for the state
2. Arguments — types, required fields, IDs bound from context
3. Execution — timeouts, errors, idempotency (your executor, not the model)
4. Result interpretation — the next model step uses the payload, not prior guesses
5. Recovery — structured retries vs blind loops. A legitimate transient failure followed by a correct retry is recovery quality, not an automatic trajectory fail.
6. Permission / safety boundaries — the model never "selects" a tool the user cannot invoke

Example task: "Find the latest sales report and summarize the top three regions."

Deterministic scorers should own selection and arguments whenever you can. Use a judge for "did the summary stay inside the file." Pair this with tool calling design: typed schemas and structured errors make evaluation possible; vague string tools make it guesswork.

Evaluating Multi-Agent Systems

Multi-agent systems add coordination dimensions on top of single-agent eval. Underlying metrics (task success, tool accuracy, safety, cost) still apply. Final task success remains required; it is not sufficient by itself when several agents acted.

Additional dimensions:

• Delegation correctness — the right specialist received the subtask
• Handoff quality — payload complete, no lost constraints or IDs (multi-agent handoffs)
• Coordination — no deadlock, livelock, or infinite supervisor ping-pong
• Duplicated work — two agents calling the same expensive tool
• Communication quality — messages that a downstream agent can act on
• Role boundaries — the writer does not silently gain cancel_order
• Final task success — the graph's outcome, not the chatty intermediate

Score the system trace (supervisor + workers) as one trajectory with labeled spans per agent. If you only score the last speaker, you will ship duplicated refunds with a polished summary.

Architecture

Vendor-neutral agent evaluation is a pipeline on top of the runtime, not a widget inside the prompt.

Diagram: Agent evaluation stack

flowchart TB
    AG[Agent] --> TR[Tracing]
    TR --> EV[Evaluation]
    EV --> DS[Dataset]
    DS --> RG[Regression]
    RG --> MON[Production]

Traces feed scorers; scorers feed the dataset; the dataset gates deploys; production sampling extends the dataset.

Evaluation itself usually splits into:

• Component metrics (prompt/model/tool/retriever)
• Tool-call evaluation
• Trajectory evaluation
• Outcome evaluation
• Safety evaluation
• Human evaluation

Those branches write to the same evaluation dataset and the same regression runner. LangGraph and similar orchestrators persist runs; tracing platforms record spans; eval libraries attach scores. The architecture does not require a particular vendor. It requires a stable run_id, span semantics, and scorers you can run in CI.

See Best AI Agent Frameworks when comparing how stacks expose traces and interrupts — those hooks are what evaluation attaches to.

Step-by-Step Flow

Diagram: Agent evaluation workflow

flowchart TD
    A[Define tasks] --> B[Build dataset]
    B --> C[Run agent]
    C --> D[Capture traces]
    D --> E[Score actions]
    E --> F[Score trajectory]
    F --> G[Score outcome]
    G --> H[Analyze fails]
    H --> I[Improve]
    I --> J[Regression]
    J --> B

Evaluation is a loop: every production or CI failure should earn a case, a scorer, or both.

1. Define tasks — user jobs, not model puzzles. Write success criteria a second engineer can apply.
2. Create the evaluation dataset — fixtures, tools, labels, forbidden actions.
3. Run the agent — freeze versions; record seeds if you sample.
4. Capture traces — model calls, tools, retrieval, HITL, errors.
5. Evaluate actions — selection, arguments, schema, permissions.
6. Evaluate the trajectory — constraints, loops, recovery, handoffs.
7. Evaluate the final outcome — task, answer, groundedness, safety, cost.
8. Analyze failures — assign a root-cause class before changing the prompt.
9. Improve the agent — prompt, tools, retrieval, rails, orchestration.
10. Run regression evaluation — the suite must stay green; new cases stay in.

Real Production Example

Agent: customer support
Task: "Find the status of order 12345 and explain why it is delayed."
Tools: get_order, get_shipment, search_kb
Policy: read-only; no refunds; do not cite other customers' orders.

Success criteria:

• Order 12345 is the only order referenced
• Status and delay reason match tool payloads
• No write tools
• Terminates within 6 model steps

Successful trace (compressed):

1. get_order(order_id="12345") → {status: "in_transit", shipment_id: "S-88"}
2. get_shipment(shipment_id="S-88") → {last_event: "weather_hold", location: "MEM"}
3. Final: order in transit; delay is a weather hold in Memphis; no KB call needed

Scores: task success 1, tool selection 1, arguments 1, groundedness 1, safety 1, steps 3.

Failed trace:

1. search_kb(query="why orders delayed") → generic peak-season article
2. get_order(order_id="1234") → not found
3. get_order(order_id="12345") → in transit
4. Final: "Your order is delayed because of holiday volume." (ignores tracker)

Scores: task partial (found order, wrong reason), tool selection 0 on step 1, argument error on 1234, groundedness 0, trajectory wasteful.

A compact scorer mixes determinism with a judge only where needed:

from typing import Any

FORBIDDEN_TOOLS = {"refund_order", "cancel_order", "email_customer"}
REQUIRED_PREFIX = ("get_order",)


def score_support_trace(trace: dict[str, Any], expected: dict[str, Any]) -> dict[str, Any]:
    calls = trace.get("tool_calls") or []
    names = [c.get("name") for c in calls]

    forbidden = sorted(set(names) & FORBIDDEN_TOOLS)
    order_args_ok = any(
        c.get("name") == "get_order" and str(c.get("arguments", {}).get("order_id")) == expected["order_id"]
        for c in calls
    )
    used_required = names[:1] == list(REQUIRED_PREFIX) or "get_order" in names
    step_count = int(trace.get("model_steps") or 0)

    deterministic = {
        "task_success": bool(trace.get("end_state", {}).get("order_id") == expected["order_id"])
        and bool(trace.get("end_state", {}).get("delay_reason") == expected["delay_reason"]),
        "tool_selection_ok": used_required and "get_shipment" in names,
        "tool_arguments_ok": order_args_ok,
        "safety_ok": not forbidden,
        "step_efficiency_ok": step_count <= expected.get("max_steps", 6),
        "forbidden_tools": forbidden,
    }

    # Judge only the user-visible answer against tool payloads, not tool names.
    evidence = trace.get("tool_results") or []
    final = trace.get("final_answer") or ""
    deterministic["needs_groundedness_judge"] = bool(final) and bool(evidence)
    return deterministic

Wire timeouts, auth, and PII redaction in the tracer, not in the scorer. The scorer should be idempotent on a stored trace so CI can replay.

Design Decisions

Common patterns

• Constraint eval — required tools, forbidden tools, max steps, must-cite shipment id
• Environment eval — assertions on DB/ticket/test suite after the run
• Pairwise agent bake-off — same cases, two prompts or models, order-swapped judges
• Replay eval — stored traces scored with new judges after rubric changes (does not re-execute tools)
• Safety suites — injection, cross-tenant IDs, write-tool temptation

Comparisons

Agent evaluation vs LLM evaluation

LLM eval is a component of agent eval (final answer, sometimes step critiques). It does not replace trajectory or tool scores. See LLM evaluation.

Agent evaluation vs RAG evaluation

When the agent retrieves, reuse faithfulness and context precision from RAG evaluation. Still score whether retrieval was the right action at all — retrieving when get_order was required is an agent failure, not a retriever failure.

Common Mistakes

1. Evaluating only final answers — lucky paths and unsafe paths look identical.
2. Relying only on public benchmark scores — GAIA/SWE-bench do not encode your tools or policies.
3. Using one metric — task success without cost, or cost without safety, produces the wrong optimum.
4. Using LLM judges without calibration — you will ship the judge's biases.
5. Ignoring tool calls — in tool-using agents, failures often concentrate in selection, arguments, or result misuse.
6. Ignoring failed trajectories — crashes and max-step stops never reach the answer scorer if you drop incomplete runs.
7. Evaluating only happy paths — injection, empty tools, and partial data are the real distribution tail.
8. Not testing regressions — prompt and model swaps need the same suite, not a new demo.
9. Measuring latency and cost separately from quality — you will "improve" quality with unbounded loops.
10. Assuming production matches the benchmark — live tickets, auth, and clock skew are different tasks.

Where It Breaks Down

Evaluation fails as a system when scores are not actionable. After a failing case, classify root cause before editing the prompt:

If you cannot name the class, you are not evaluating — you are staring at a scoreboard.

Non-determinism also breaks naive CI: a 1-point flake is not a regression. Use enough cases, freeze temperature where you can, and gate on deltas with tolerance — the same discipline as AI evaluation CI gates.

When NOT to Treat Scores as Ship Criteria

Skip heavy agent evaluation machinery when:

1. There is no loop — a single LLM call with no tools is LLM evaluation, not this guide.
2. The path is a fixed workflow — assert each node; do not pretend the DAG is an agent. See workflows vs agents.
3. You cannot observe actions — add tracing first (observability).
4. Labels are noise — thumbs-down without a rubric will not gate deploys.
5. The judge is uncalibrated on safety — do not auto-ship on a safety score you have not checked against humans.

A small, brutal suite of real failures beats a large unmaintained benchmark export.

Running in Production

Production evaluation is a loop, not a dashboard screenshot:

Production traces → sampling → automated evaluation → human review → failure analysis → dataset update → regression suite.

Best Practice

✅ Best Practices — Pin dataset + prompt + tool schema + model ID in CI. Drop incomplete traces into the suite instead of excluding them. Calibrate judges on a frozen human set. Never let a single "helpfulness" score own a write-capable agent.

Important

Production traces often contain personal data. Minimize what you store, restrict who can read eval queues, and keep legal/compliance review with your own counsel — this guide is engineering practice, not legal advice.

Regression evaluation belongs in the same pipeline as unit tests: a change to models, prompts, tools, retrieval, memory, orchestration, or guardrails re-runs the agent suite. That suite is how you change agents without relying on a stage demo.

Related Guides

• AI Agents · Tool Calling · Agent Loop Lab · Tool Calling Lab
• Observability · AI Evaluation · LLM Evaluation · RAG Evaluation · Prompt Evaluation
• Guardrails · Human-in-the-Loop · Multi-Agent Systems · Multi-Agent Handoffs
• Rankings: Best AI Agent Frameworks · LangGraph vs CrewAI
• Tools: LangGraph · LangChain · Langfuse · DeepEval · Braintrust · Phoenix
• GitHub: Top AI agent repositories (2026) · Agents on GitHub

If you understood this topic, read next:

Diagram: Learning path for agent evaluation

flowchart LR
    A[Agents] --> B[Eval]
    B --> C[Loop lab]
    C --> D[Tools]
    D --> E[Rails]
    E --> F[HITL]

Build the loop, measure it, then constrain writes with rails and human gates.

Prerequisites: AI Agents · Tool Calling · AI Evaluation

Next topics: Observability · Guardrails · Human-in-the-Loop · LLM Evaluation

Estimated time: 60 min · Difficulty: Advanced

Interview Questions

1. How is evaluating an AI agent different from evaluating an LLM response?

   • Expected: agents produce trajectories and side effects; score outcome and path (tools, recovery, policy), not only the final string.

2. What should an evaluation case contain?

   • Expected: task, environment/tools, success criteria, optional reference and action constraints; sourced from real failures, not only happy paths.

3. Why is the execution trace often the primary artifact for agent evaluation?

   • Expected: for many tool-using systems, spans for model/tool/retrieval let you score steps, trajectories, and outcomes; component tests and end-state oracles still matter without a full trace.

4. When would you use LLM-as-a-judge vs deterministic checks?

   • Expected: schema/tool names/end-state assertions first; judges for faithfulness and open-ended quality, calibrated against humans, never as unaudited ground truth.

5. How do you evaluate tool-using agents?

   • Expected: selection, arguments, execution, interpretation, recovery, permissions — with a concrete example like order lookup vs knowledge-base guess.

6. What extra dimensions appear in multi-agent evaluation?

   • Expected: delegation, handoff payload quality, duplication, deadlocks, role boundaries, plus still-required task success.

7. How should evaluation work in production?

   • Expected: sample traces, auto-score, human-review high-impact slices, classify failures, add cases, re-run the regression suite; mind PII and cost.

8. A task success rate went up and p95 latency doubled. Did the agent improve?

   • Expected: no single metric; report quality with step count, cost, and safety; investigate loops and redundant tools.

Key Takeaways

• Agent evaluation measures what was produced and how the run got there.
• For many agent systems the execution trace is the primary artifact; scorers attach to spans, trajectories, and outcomes. Component tests and end-state checks still apply.
• Deterministic checks own tools and state; judges and humans own residual quality and safety.
• Offline suites are regression tests; online sampling keeps the suite honest.
• Failure classification turns a score into an engineering change.
• Guardrails and HITL are both controls and eval signals.
• Do not confuse benchmark leaderboards with product readiness.

FAQs

How do you know whether an AI agent actually works?

Define success criteria for the user's job, run the agent in a known environment, capture the trace, and score both the end state and the trajectory. Fluency of the last message is only one signal.

Is agent evaluation the same as LLM evaluation?

No. LLM evaluation scores generated text. Agent evaluation includes that score when there is a final answer, plus actions, tools, environment state, and process quality.

Can I use only public agent benchmarks?

They are useful for research comparison and model shortlisting. They do not encode your tools, data, or policies. Product gates need your own cases.

How large should the evaluation dataset be?

Start with the failures you already have, even if the initial set is small, and grow it as new intents and failure modes emerge. Coverage of failure modes matters more than raw count.

How do I score non-deterministic valid paths?

Use constraints and end-state oracles instead of a single gold action sequence. Allow multiple valid tool orders when they are equivalently safe.

Should I drop traces that never produced a final answer?

No. Timeouts, max-step stops, and tool crashes are failures. Excluding them inflates success rate.

How do I evaluate safety without a legal review process?

Engineers can test forbidden tools, injection fixtures, and cross-tenant IDs in a sandbox. Compliance, retention, and privacy programs are organizational — this guide does not provide legal advice.

What is regression evaluation for agents?

Re-running a versioned suite after changes to models, prompts, tools, retrieval, memory, orchestration, or rails — the same role unit tests play for deterministic code.

References

• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)
• AgentBench: Evaluating LLMs as Agents (Liu et al., 2023)
• GAIA: a benchmark for General AI Assistants (Mialon et al., 2023)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
• SWE-bench
• OpenTelemetry GenAI semantic conventions

Further Reading

• AI Evaluation — layered eval, golden sets, CI gates
• LangGraph documentation — persistent runs and interrupts as eval hooks
• OpenAI Agents guide
• Anthropic: Building effective agents
• Top AI agent GitHub repositories (2026)
• AI agents on GitHub