- 01. Why standard outcome evaluation breaks for agents -- and the math that makes this concrete.
- 02. What trajectory evaluation is, how it differs from pass/fail scoring, and when each belongs in your eval stack.
- 03. The metric families that actually capture agent quality: tool correctness, argument correctness, step efficiency, plan adherence, and partial credit scoring.
The Story
Consider a procurement automation agent given a task: find the best-priced vendor for a standard supply order and place it. It completes the task -- order placed, vendor selected, price looks right. The team scores it as a pass. Three weeks later, on a similar order with a currency conversion edge case, the same agent selects the wrong vendor, and finance catches a significant pricing error.
When the team pulls the trace on both the original passing run and the failing run, they find something uncomfortable: the paths are almost identical. The first run had gotten lucky. The agent had queried a stale vendor catalog, skipped a currency normalization step, and escalated to human review based on a threshold condition that shouldn't have fired. Every one of those problems existed in the passing run. The outcome just happened to be correct.
This is the core problem of agent evaluation: you can measure whether the final answer is right without ever seeing whether the reasoning that got there is sound.
Consider a customer support agent that handles account-related questions. A user reports: "I asked your bot to reset my password and it told me my account was suspended. It isn't." The agent had called an authentication-check tool, received an HTTP 503 error, and then confidently told the customer their account was suspended -- generating a plausible, detailed explanation for a fabricated status. This is agent gaslighting -- the third face of fake success from the taxonomy introduced in Topic 2 -- where the agent produces a confident, coherent reason for its behavior that's based entirely on a tool error it misclassified as content.
Consider an enterprise IT helpdesk with three specialized agents: a triage agent, a systems-access agent, and an escalation agent. The agent pipeline "completed." The user spent 40 minutes re-explaining context the system already had. The failure wasn't in any individual agent's output -- it was in what got dropped at each handoff boundary.
Three different systems. Three different ways agent evaluation can fail if you only look at whether the final output was correct.
The Core Idea
Agent evaluation is harder than RAG evaluation or conversation evaluation for one structural reason: agents execute sequences of actions where each step's correctness depends partly on prior steps, and a correct final answer can emerge from a broken path.
The math makes this concrete. If an agent must execute 10 sequential steps and each step succeeds with 80% probability, the end-to-end success rate is 0.80^10 = 10.74%. Not 80%. Not even 50%. What looks like an 80% agent is actually delivering end-to-end reliability below 11%.
The solution is layering two different evaluation types: outcome evaluation (did the agent accomplish the goal?) and trajectory evaluation (did the agent take the right path?). Trajectory evaluation doesn't replace outcome evaluation. It runs alongside it. Outcome tells you what happened. Trajectory tells you whether what happened was repeatable.
The metrics that actually matter for agent evaluation fall into distinct families. Task completion is the binary or graded outcome metric. Tool correctness measures whether the agent selected the right tool for each step. Argument correctness asks: given the right tool, did it pass the right arguments? Step efficiency measures whether the agent accomplished the task in a reasonable number of steps. Plan adherence and plan quality extend the trajectory model to reasoning structure.
Partial credit is a first-class design choice, not a consolation prize. In a 10-step task, an agent that completes 9 steps correctly and fails on step 10 is not equivalent to an agent that fails on step 2. Binary pass/fail scoring collapses this distinction.
Where This Hits in Production
Regulated industries treat handoff boundaries as governed interfaces. In healthcare, financial services, and legal tech, a multi-agent pipeline that degrades context at handoff boundaries isn't just a quality problem -- it's a compliance problem. The handoff packet must be auditable: what did agent one know, what did it pass, and on what basis did agent two act?
The four LangSmith trajectory matching modes matter more than they sound. Strict, Unordered, Subset, and Superset each encode your assumptions about agent flexibility. A compliance agent might use strict mode. A research agent might use subset mode. The choice is not technical -- it's a product decision about how much autonomy the agent is allowed.
Mode = product decision, not technical setting
Encodes your tolerance for agent autonomy.
Set it wrong and the eval lies to you.
Quick picks
Connecting the Dots
Agent evaluation is where the autonomy spectrum becomes a hard requirement rather than a conceptual tool. At Level 3-4, outcome evaluation is probably sufficient. At Level 5-7, trajectory evaluation isn't optional. A Level 6 agent that arrives at correct answers via broken reasoning paths is an incident waiting to happen at scale.
Teams that invest in trajectory evaluation infrastructure early -- span-level tracing, reference trajectory libraries, harness-layer tooling -- accumulate an asset that takes months to build and is hard to replicate quickly. The reference trajectory library is institutional knowledge. Every failure you catch and add to that library makes the eval suite better.
Common Mistake
Treating trajectory evaluation as trajectory matching.
Strict step-by-step comparison against a reference is appealing because it's precise and automatable. It's also brittle. If your agent gets genuinely smarter -- finds a more efficient path, uses a tool combination the reference didn't anticipate -- strict matching fails it. You'd be penalizing improvement.
The second version of this trap: evaluating agents only in happy-path scenarios. Most eval suites test the cases where tools work, APIs respond, and inputs are well-formed. A production agent will hit every one of those failure conditions.
Run both: outcome scoring to know whether the agent succeeds, trajectory evaluation (with flexible matching modes) to understand why.In Practice
Layer 1 -- Outcome scoring. Start with binary task completion on a golden dataset of 50-100 representative tasks. This gives you a baseline. It doesn't tell you why you're at that baseline.
Layer 2 -- Span-level tracing. Instrument your agent to emit spans for every tool call and model call. This is the prerequisite for trajectory evaluation.
Layer 3 -- Reference trajectories. For your highest-stakes task types, build reference trajectories. Start with 10-15 per task category. The reference library is the highest-leverage investment in agent eval.
Layer 4 -- Tool and argument scoring. Add tool correctness and argument correctness metrics to your harness so that agent deployments that degrade tool selection accuracy fail the build.
Layer 5 -- Error recovery testing. Add adversarial scenarios: tool failures, malformed API responses, ambiguous inputs. This is where you surface the agent gaslighting pattern before users do.
The full stack takes weeks to build, not days. Most teams should start with Layer 1 and Layer 2 in parallel, then add reference trajectories for the task types that matter most.