What Can You Actually Evaluate on an Agent Today?

What Can You Actually Evaluate on an Agent Today?

You have an agent. It mostly works. Then someone asks whether it is any good, and you realize you have no honest way to answer. This is a walk through the machinery that does answer it, and a confession of every place we caught the machinery lying. The punchline, up front: evaluation is not a number. It is a question, and a place.

The afternoon someone asks if it is good

Here is the scene, because you have lived it. The agent is in production. It handles most of what you throw at it. Then a stakeholder, or your own conscience, asks the flat question: is it good? You open the traces. You scroll. You feel the specific dread of a person holding a great deal of data and no way to turn it into a verdict.

The dread is justified, because “is it good” is not a question. It is a mood. Good at what, judged how, seen from where. It bundles a dozen separate questions under one word and then dares you to answer all of them with a single number. You cannot, and the rest of this is what you do instead.

We will build the answer in the order you would actually discover it. First the thing everyone reaches for, and why it betrays you. Then the small, almost embarrassing fix. Then the real machine underneath, which is where the research lives, and where we will be precise about what we measured versus what we merely built.

Act one: the number that means nothing

The first instinct is universal. You take the output, hand it to a capable model, and ask it to rate the result from 1 to 10. The model returns a 7.

Seven what. Seven out of a scale it invented on the spot, for a criterion you never wrote down, calibrated against nothing. Run it again at the same temperature and it returns a 6, then an 8, because you asked a question with no defined answer and it politely handed you a number-shaped opinion. You now own a metric. It drifts. It has no unit. And its single most dangerous property is that it looks like measurement.

So you do the responsible thing and add a second judge to cross-check the first. It agrees. You relax.

You should not have. Here is the first finding, and it is the one that should change how you read every agreement number you will ever see. The question we wrote down was blunt: is a second opinion actually independent, or is it one instrument counted twice? What we found: two draws from the same model, agreeing, are not corroboration. They are the same bias charged to your account twice. In one real case a ship gate agreed with its own reference at a correlation of +0.999, and it was not two witnesses, it was one computation wearing two badges. Worse: under the mapping the gate actually used in production, the two agreed on exactly 1 run out of more than 6,000. And you cannot escape this with a threshold, because a genuinely independent pair of methods landed at 0.958 while the two same-model draws sat at a flat 1.0. Agreement magnitude cannot tell them apart. A high number between two things that are secretly the same instrument is not certainty. It is a mirror, and it shines brightest precisely when it is worth the least.

That is the trap most of the industry is standing in: a scored, model-judged, self-agreeing pipeline that produces confident numbers and measures nothing. Now the fix.

Act two: stop scoring, start asking

The trick is so small it feels like cheating. Stop asking for a score. Ask a question that is true or false.

Not “how good was the email,” which has no answer. “Did it get a manager review before it sent,” which has exactly one. That is a criterion: a single question about a run that resolves to true or false. The piece of machinery that answers one criterion for one run is a check, and a check returns one bit. Not a score. A bit.

We spent real effort stress-testing this unit, because it is the foundation and a soft foundation ruins everything stacked on it. The question we wrote down: what is the actual unit of an evaluation? What we found, after 10 rounds of adversarial review trying to break it: the unit is one criterion, one bit, and confidence never gets to flip that bit. Your judge may report how sure it is. It may not vote with it. A false at confidence 0.55 is still false. Confidence decides which verdicts a human re-reads. It does not round a failure up to a pass because the model felt strongly.

And the one refinement that survived the review is the part people skip: you cannot enforce this with a schema. A schema stores fields, it does not police control flow. Two systems can log the identical verdict-and-confidence shape while one of them quietly lets a 0.49 turn a fail into an abstain three functions downstream. The rule lives in your code path and your tests, or it does not live at all. A repository that validates the shape of your verdicts has not enforced the standard. It has framed it and hung it on the wall.

Now you have honest units. One question, one bit, no drift. Which raises the question the rest of the field mostly walks past: where does the evidence to settle a criterion actually live?

The reveal: it is a question and a place

Here is the whole machine.

An evaluation has two independent axes, and almost every confused eval conversation is someone welding them into one. The first axis is HOW you judge: a binary check is one option out of about eight, alongside reference-matching, pairwise preference, population statistics, adversarial probing, and, underneath all of them, calibrated human audit. Pick the method the question deserves. The second axis, the one nobody instruments properly, is WHERE the evidence lives. Any fact that would flip a criterion is sitting in exactly one of four places. Name the place, point the method at it, and the fog clears.

We turned each of those four places into a research question and went looking. This is the part worth your attention, because it is where the value is, and where we are careful to tell you what we measured versus what we merely constructed.

The four layers, as four questions we asked

One warning before the tour: these are not four separate grades you average into a 3.7. There is one criterion, and the fact that settles it happens to live in one of four rooms. The layers are where you look. The criterion is what you ask.

Outcome: what it finally produced

The question we wrote down: can you tell whether the agent succeeded, without reading how it got there? An outcome check reads the end state and refuses to look at the path, which is its superpower, because it survives you rewriting the agent’s internals. If the artifact is wrong, outcome says so, and it does not care how the wrongness arrived.

But there is a prior question, and it bites. Before you can judge an outcome, you have to ask whether your recorded runs can even separate “did it succeed” from “what did it do.” What we found, running that separability check over 6 real trace sets: two could answer an outcome question, one was partial, and three structurally could not. The partial one is the cautionary tale. It shipped with an external label on every row and a second, independent verifier, except the verifier agreed with the label almost perfectly, which means it was not a second opinion, it was a photocopy. Two verifiers on paper. One and a half in fact. Build an outcome scorer on a trace set that cannot separate outcome from path and you have built a trajectory check in a costume, and it will hand you a confident number that means nothing.

Where the trace set does support it, a second question follows: can an instrument judge the end state with the path physically barred in code, and does a semantic judge recover the cases a crude keyword rule fumbles? What we found: on a small fault-injected trace set, the end-state instrument agreed with the labels 0.979 of the time, and the one case it missed was a run where the agent correctly refused to act, which the semantic judge caught and the keyword rule did not. The fence rides along with that number, because it always must: 48 units, synthetic faults, a characterization of mechanics and not a field-accuracy claim. A number without its fence is an overclaim.

Trajectory: what it did, in order

The question: what did the agent do, step by step, and which part of it must a human actually read? This is the layer that knows where the run first went off the rails, and where a fix usually gets routed, because it can point at the exact step the sequence broke.

The hard part is not judging a trajectory, it is reading them at all when you have more traces than a team could open in a year. So we asked the uncomfortable version: does the failure hide in the region your automatic matcher gave up on? What we found should retire the phrase “top failures by frequency” from your vocabulary. The failure we were hunting ran about 30 times more often inside the discarded, unmatchable remainder than in the neatly classified region, and a scan of the 50 most frequent clusters surfaced 2 of the several hundred clusters where it actually lived. The remainder is not noise. It is the crime scene, and the standard sampler is a flashlight pointed at the one place the body is not.

Two more trajectory questions, briefly, because the answers are useful. How much of a trace can a reviewer skip without losing the evidence? About half, if you read the divergences from expected behavior plus every irreversible action and its guard window and skip the stretches that behaved. And which call should a human look at first? The one that took an irreversible action without its guard in front of it, which a simple ranking floats to the top. What trajectory cannot do, ever, is tell you whether the answer was any good. A flawless, rule-abiding path reaches a confidently wrong answer all the time.

Tool outputs: what the world said back

The question: what did the world actually say when the agent reached into it, and are we even listening? Every API call, every search, every send returns a verdict from reality, and this is the layer that catches the exact moment a confident-looking run turns out to be fiction, because the agent built its next three steps on a result that was never really there.

Then the finding that is less a discovery than an indictment, of nearly everyone. We asked: does the agent proceed as if a failed tool had not failed? We could not answer it, and the reason is the whole point. The detector runs fine. It had nothing to read, because the trace recorded which tools were called and threw away whether they worked. The failure happened, the API screamed, and the log wrote down “called send” and dropped the part where reality said “no you did not.” The success signal was not missing from the universe. It sat one layer upstream in the raw responses, present on most of them, and was simply never carried into the index the checks actually read. That is not a research problem. That is a logging change, and it is the cheapest high-value fix on this entire page: record the success or failure of every tool call and this whole layer wakes up. It is also the change that decides whether a root-cause pass over your traces can reach this layer at all, because a descent cannot walk a step nobody recorded. The neighbouring question, how much of your tool surface any check even watches, tends to answer itself, and the answer is “less than you would guess.” That is a scope number, not a quality one. Watching a tool is not the same as watching it well.

Context: what it actually had in front of it

The question: what did the agent actually have in its window at the moment it decided? This is the layer that separates two failures that look identical and have opposite fixes. Either the deciding fact was never in front of the agent, which is a retrieval problem you fix in the pipeline, or it was right there and the agent walked past it, which you fix in the model or its instructions. Confuse the two and you will keep shipping the wrong fix and wondering why the number will not move.

What we found: the reliable half of this today is exclusion. Running a presence census over thousands of real decisions, for a large share of them the category the answer would have lived in was simply not in the window at all, which lets you rule out “it was ignored” completely and march straight at retrieval. The other direction, proving a value was present and got ignored, needs a record of what each decision required, and most runs do not carry that contract. So we lean on the exclusion we can prove and treat “present but ignored” as a claim you have to earn. Honest beats complete.

Reading four layers across thousands of runs is not something a person does by hand, which is the job the diagnosis pass exists to do.

03Mutagent Use CaseDiagnostics A failure-mode scan that finds root causes, not a wall of logs. Every finding is traced to a step and an origin, and ranked by impact. IN production traces    OUT ranked root causes · failure index Explore

The assembly: an eval suite for an email agent

Enough principle. Here is the shape of a real suite for an agent that drafts and sends email, each row a criterion, its method, and the layer that holds its evidence.

The precondition sits above the table, because it is a gate and not a criterion: can these runs even separate the outcome from the path. If not, the outcome rows are not askable, and you want to know that before you write them. Then the criteria. Goal met, judged on the final send alone: a binary check on the outcome layer. The draft went through review before sending: trajectory. It did not fire the irreversible send without its guard: trajectory again, and the row you guard hardest, because a send that skipped review completes cleanly and looks healthy to every other check in the suite. The send actually went through instead of silently failing: tool outputs, and therefore exactly as answerable as your logging is honest. The information each decision needed was in the window: context.

Two rows you cannot write yet sit greyed at the bottom, and they are not mysteries. “Did it notice the tool failure” waits on that one logging change. “Did it have what each decision required” waits on a declared contract. They are the tool-outputs and context layers, the two most suites under-read, which is not a coincidence. The layers nobody logs are the layers nobody checks.

Writing that table is the work. Calibrating the judge that fills it in, against someone who actually knows the domain, is the part that decides whether the table means anything.

02Mutagent Use CaseEvaluate Builds the evals and a calibrated judge that scores every change, aligned to your domain expert before it gates anything. IN dataset · expert labels    OUT calibrated judge · criteria Explore

What “trustworthy” actually costs

Anyone can produce a number. A model will hand you one for free, all day, in whatever range you ask for. The expensive thing, the only thing that separates evaluation from astrology, is knowing which of your numbers have met reality and which have only ever met each other. Every instrument here ships with its fence bolted on, not out of a love of disclaimers, but because a number that will not tell you where it stops being true is not a measurement. It is a rumor with a decimal point.

That is the discipline, and it is smaller than it sounds. Not a metric. A question that can come back false, the one place its answer actually lives, and the fence that says how far that answer travels. Get those three right and you stop asking whether your agent is good, because you will already know, in specific and defensible detail, exactly what it did.

---

Related reading: why a moved score is an alarm and not a cause, what happens when the judge grades itself, and the benchmark behind the independence numbers.

In the product, these are two jobs: building the eval suite and the judge that scores it, and finding the root cause once a score moves. More on how we build them at mutagent.io.