44% vs 85%: your first cross-agent eval measures your eval harness, not the model

Two coding agents — Codex and Claude Code. The same 417 tasks, an early version of the suite now public at UiPath/skills. Codex: 44.1%. Claude Code: 85.3%.

A forty-one point gap is not a nuance, it is a verdict — the kind that gets screenshotted into a decision meeting: we evaluated both, it wasn’t close.

We audited it before publishing anything. Seven sub-agents in parallel, one instruction each: for every failed task, name the mechanism and cite the run record. That covered 227 of the 228 failures, no sampling; the rest errored out without producing a verdict to audit.

Eleven percent of them were the model. The rest were our eval harness — the sandbox, the environment, the grader — and our own over-strict checks. The honest read is not “Codex is half as good,” it is “we spent a suite’s worth of agent time measuring the machine doing the measuring.” Everything the audit blamed on the harness has since been fixed in Coder Eval; the specifics are below.

The one-line version

Most of it was knowable before the agent started. Run the grader’s commands first; abort if they fail:

# Runs in the sandbox after setup, before the agent starts.
pre_run:
  - command: "uip --version" # the grader shells out to this later
    timeout: 15
  - command: 'test -n "$REGISTRY_TOKEN"' # and reads this
    timeout: 5
  - command: "node --version" # informational, never fatal
    timeout: 15
    fail_on_error: false

A non-zero exit aborts with FinalStatus.ERROR instead of scoring. The failing case:

Running pre-run command: test -n "$REGISTRY_TOKEN"
Evaluation failed: Pre-run command failed (exit 1)
Task finished: status=ERROR duration=1.4s score=0.000 iterations=0

iterations=0 is the point. No agent turn was spent, and the task landed in ERROR, not in the denominator of a pass rate. A broken environment announces itself instead of impersonating a weak model.

Attribution, four buckets

Every failure went into exactly one of four buckets. The taxonomy transfers to any suite:

The ambiguous bucket was not a dumping ground — it was one cluster of 30 tasks failing downstream of a single broken tool. Reclassifying it on the evidence puts roughly 89% of all failures outside the model’s control.

The same run under each accounting:

What actually broke

Eight mechanisms explain nearly all of it, and none of them are exotic:

• One tool, one bad backend — 106 of the 228 failures. Codex’s only file-create/edit tool was a single API-backed operation, and that endpoint was erroring. Sandbox policy forbade an unapproved shell fallback, so the edits never landed. One flaky dependency, just under half the suite.
• An expired package-registry token. Every plugin install returned 401/403, so the tooling the tasks were about was never present — wiping out one skill’s tasks entirely and most of another’s.
• Write-path divergence. When the agent did fall back, it wrote into its own working directory while the grader looked in the run’s artifact directory. Correct content, wrong place, zero score.
• An activation check that knew only one agent’s shape — 30 tasks. skill_triggered recognized Claude Code’s explicit Skill call, which Codex does not have. Those 30 failed regardless of what Codex did; Claude Code passed all 30.
• Command matching that graded keystrokes. command_executed regexes the command text over its first 2,000 characters — a deliberate ReDoS bound, and a trap: quote-escaping, a literal ID where the agent used a variable, or a different flag order all read as “never ran it.”
• Plus the product’s own auth expiring mid-run (~10), suite drift including a check that contradicted the documentation it tested — a test no correct agent could pass (~5), and coverage gaps (5).

The first is a design lesson, not a vendor complaint: mapping every file mutation onto one backend call makes it a single point of failure for the whole eval.

One more thing turns those mechanisms into a forty-point gap: a task’s verdict is a strict AND over every gating criterion, so a ten-criterion task with nine passes and one unfair failure scores zero. A single miswired check appearing across a hundred tasks costs you a hundred tasks, not a hundred criteria. (The escape hatch is weight: 0 — informational criteria are excluded from the score and the gate, so a check you suspect is agent-specific can be demoted rather than deleted. Demote first, argue later.)

Every harness-level item on that list is now closed in Coder Eval. skill_triggered is agent-agnostic — it credits an explicit Skill call and a skill’s files read off disk, so Claude Code, Codex and Antigravity score the same behavior identically. Pre-flight commands are a first-class task field, and the write path is defined by the task, not by wherever a fallback lands. Popular coding agents can now meet on the same suite with most of the comparison bugs gone — the precondition for a cross-agent number meaning anything at all.

The inverse law of gap size

The heuristic the audit produced, and the one we reach for first. Rank the per-skill gaps between two agents. The bigger the gap, the less likely it is to be real.

It reads backwards until you think about the generating process. Genuine capability differences are graded and modest — a model is a bit worse at something, so it fails the harder instances. Infrastructure failures are binary and total: the tool installed or it did not, so every task depending on it fails together. A 100-point gap is not evidence of a chasm in ability. It is evidence that something was switched off.

That gave us a decisive test, close to a natural experiment: did this agent’s tooling actually install for this skill? Where it did not, the failures were harness. The one tool that installed cleanly is where the genuine failures clustered.

The genuine residual was specific: given a query that returned no rows, Codex piped the empty variable into the next call and got a 500 back; Claude Code noticed, broadened the query, and continued. That is not “worse at authorization” — it is weaker adaptive persistence across a multi-step chain, and it would never have been visible under the 41-point headline.

The kicker: the grader graded itself

The detail that changed how we build tasks: the grader’s own verification commands failed in the same broken session. The checks meant to adjudicate the agent’s work could not succeed either — and reported that as the agent’s failure.

An eval that cannot verify its own preconditions is not measuring the agent. The checklist, cheapest first:

1. Same OS, same shell for every arm. Different machines is not a detail; it voids the comparison.
2. Run the grader’s commands before the agent starts — the pre_run block above. Prove the CLI, the token and the network path exist while failing is still free.
3. Expiring credentials are pre-flight items, not assumptions. Ours expired mid-suite, twice.
4. Audit every criterion for agent-specific shape. Any check keyed to one agent’s tool names, paths, or output format is a harness artifact waiting to happen. Ask: could a correct answer from a different agent fail this?
5. Assert on the artifact the task is about, in the directory the task defines — not wherever a fallback landed.
6. Read the run record before the score. A whole skill at zero is an infrastructure alarm, not a capability finding.

Points 4 and 5 are the idea running through grading outcomes rather than keystrokes: assert what the artifact does, not what the agent typed.

What this method does not do

A hand-audit of 227 failures does not scale, and we are not proposing it as routine. It is a one-time instrument for a first cross-agent run — where you have no baseline and every number is equally plausible. After that, the pre-flight and a fixed criteria set carry it.

It also has a bias worth naming: attribution is a judgment call, and the incentive runs toward charity — it feels sophisticated to call a bad result a harness artifact. Two guards kept us honest. Every label cited a mechanism from the run record, and the tell was mechanical (did the tool install?) rather than interpretive. The 22% we could not decide stayed ambiguous instead of being distributed to taste.

An independent check: even in the subset where the broken write tool cannot be the cause — 125 failures reachable through the shell alone — the ordering repeats. Install failures 55, tool side effects 25, auth 8, strictness 5. A genuine core of 15 to 18 tasks, out of 417.

The claim to check on your own suite

If you have a cross-agent comparison in a spreadsheet, this is testable this afternoon, without re-running anything:

Rank your per-skill gaps. Pull the run records for the three largest and the three smallest. You will find more infrastructure in the large ones. If the biggest gaps instead show a graded mix of near-misses and partial credit, the difference is probably real — and now you know it for a reason, not because a number was large.

The stronger version costs one more run: fix what the pre-flight flags, then re-run. If the gap collapses, it was your harness. If it holds, you have a finding that survives the first person who asks how you know.

That is the test we ran on ourselves. We fixed the harness and re-ran: 44% became 76%, against 82% for Claude Code — the gap collapsed. What remained was mostly Codex following instructions less literally while still reaching the goal, a criteria-design problem rather than a capability one, and a separate post.

Coder Eval is open source (Apache-2.0); pre_run, per-task run records and criteria that run outside the agent’s environment are covered in the task definition guide, with how it compares to public benchmarks on its own page. Starting from zero? The introduction is the shorter road in.

uv tool install coder-eval   # or: pip install coder-eval