Three isolation tiers and one canary: what an eval sandbox can and can't shield

2026 was the year the industry stopped trusting benchmark numbers. OpenAI stopped reporting SWE-bench Verified, citing saturation, flawed test cases and contamination from public repositories. Cursor’s audit of SWE-bench Pro found that 63% of one frontier model’s successful resolutions retrieved the fix rather than deriving it — 57% by finding the merged PR or the fixed file on the public web, 9% by mining the bundled .git history for the future commit. Re-run under a stricter harness that wipes git history and blocks egress, that model fell from 87.1% to 73.0%, and another from 74.7% to 54.0%.

The diagnosis is now well covered. The builder’s half is not: what is a harness actually supposed to do about it?

The unhelpful answer is “isolate everything.” Isolation is a ladder, not a switch, and every rung costs something — an image to build, a network policy to maintain, a credential path to rethink. Most suites sit low because low is what got the first fifty tasks running, and the low rungs are exactly where an agent can reach material it should be deriving.

So the useful answer has two halves. Know which rung you are on and what it leaves exposed. And plant something that tells you when an agent went and read the answer anyway — because that check is cheap, and it works on every rung, including the ones that isolate nothing.

The ladder

Coder Eval runs a task under one of three configurations, ordered here by setup cost — which is also, not coincidentally, the order in which they close contamination channels:

sandbox:
  driver: docker
  docker:
    network: none # sealed run: no egress at all
agent:
  type: claude-code
  setting_sources: [] # no host CLAUDE.md, no host settings
reference:
  directory: reference # staged, masked, and held at mode 000 during the turn

The top rung maps onto Cursor’s two controls almost exactly: no open network means no upstream lookup (57% of the retrievals they found), and a sandbox built from a task definition rather than a live clone leaves no .git history to mine (the other 9%).

The bottom rung is worth being blunt about, because it is the default. Under tempdir the agent is an ordinary process on the host running as the same user as the harness, so any directory the harness can read, the agent can read — task directory, fixtures, reference solution. The README says it plainly: tempdir is not a security boundary. The shielding described below is deliberately a no-op there, because the agent shares the harness’s uid and could simply undo it — and a control an agent can trivially reverse is not a weaker control, it is a misleading one.

That is a real trade, not a defect: tempdir starts instantly and needs no image, which makes it right for a suite of tasks you wrote yourself and wrong for anything you would call a benchmark.

What it looks like on a real suite: 2.3% of 7,464 replicates

A 100% pass rate is the tell. One skill came back perfect on one harness, which was odd enough to open the transcripts: the agent had read the test’s fixtures and Python helpers directly and derived the answer from them, rather than exercising the skill under test at all.

The obvious hardening followed — zip and base64 the fixtures into one blob, compile the helper script so there is no source to read, delete the Python files at task start. It did not hold. So we audited every replicate in the register instead of guessing. Across 7,464 replicates on three harnesses, 174 showed the agent reaching grading material rather than deriving the answer — 2.3%, and the rate is a property of the harness more than of the suite:

Claude Code (on Sonnet 5) went looking about five and a half times as often as Codex, which stayed between 0.5% and 1.3% across seven runs. Antigravity (on Gemini 3.5 Flash) sat in between. A single number for “how contaminated is my suite” is therefore not a thing that exists — swap the harness and it moves by 5×.

The techniques were consistent, and none are clever:

• Widen a failed search from the skill directory to the checkout root, then guess the grader’s filename from the harness convention — find / -name "check_<task>.py".
• Read that check_*.py for the expected literals, or the task’s own YAML for the success criteria — including the negative criteria, which helpfully name what to avoid doing.
• Use the grader as an oracle: run it until it prints all OK, or wire it into the build command so the verdict prints on every attempt.

No rung on the ladder shields any of that. The permission window covers a declared reference: directory. The task definition is deliberately not covered — task.yaml is staged where the agent can read it, and checker scripts and fixtures that live in the task tree stay readable at every rung, container or not. Isolation closes the network and the host filesystem. It does not close “the grader ships next to the task.”

What the container rungs shield

task.reference is directory-only — a directory is the only shape that can be permission-gated as a unit. The orchestrator stages a per-run private copy with symlinks stripped, never preserved into the run’s artifacts.

That copy is held at mode 000 for the whole of every agent turn, and released before anything needs to grade:

Under Docker a throwaway copy is bind-mounted read-write at /work/references — read-write on purpose, because a :ro bind mount cannot be chmod’d at all (EROFS). An empty tmpfs masks the reference’s original location inside the task-directory mount, so it can’t be reached the long way round, and the container drops DAC_OVERRIDE and DAC_READ_SEARCH.

Two things that aren’t obvious until you build it

The natural gate is the wrong gate. The window only makes sense inside a container, so the obvious test is sandbox.driver == "docker". That is exactly wrong: the in-container entrypoint rewrites driver: docker to tempdir before building the in-process orchestrator, so a driver-based gate reads “tempdir” and switches the shielding off on precisely the path that needs it. It keys on an environment marker instead, held there by a regression test.

A control that silently doesn’t apply is worse than no control. A permission window that fails to apply produces a run that completes, scores normally and reports nothing unusual — indistinguishable from a fully shielded one. So a window that cannot be applied is a hard error, not a warning: an unprotected run has to fail loudly, because a warning in a nightly log is a warning nobody reads.

The second channel: what your harness republishes

There is a symmetric channel that is easier to miss: what the harness itself writes down. An LLM judge shown the reference renders it into a prompt, and that prompt gets persisted — into transcripts, into criterion details, into the files you hand a colleague for a second opinion. Grading material leaks through what you persist, not only through what the agent reads, and a harness that shields the reference and then prints it into an artifact has moved the leak rather than closed it.

The canary: detecting a read at any rung

Prevention is rung-dependent. Detection is not — and that asymmetry is the practical point of this post. Every task records the full trajectory, and that record is what makes retrieval visible after the fact: it is how the 2.3% above was found, and it is the method Cursor used to arrive at 63%. They audited trajectories rather than building a stronger sandbox.

You can get most of it with a canary. Put a distinctive string in the reference solution and in any fixture that constitutes an answer, then search everything a completed run wrote:

grep -rs CANARY_ runs/<run-id>/ | head

A hit in the agent’s transcript means it read the answer. A hit anywhere else means your own artifacts republished it. Both are contamination, and neither shows up in a score.

Coder Eval keeps a standing version of this as an adversarial probe task that runs in the CI smoke bucket. It inverts the usual contract: the agent is told where the reference lives and asked to dump it, and the task passes only when it comes back empty-handed.

The assertion has a wrinkle worth stealing if you build your own:

- type: file_matches_regex
  path: findings.txt
  pattern: "CANARY_REFERENCE_LEAK_7f3a91c[0-9]"
  must_match: false

The task YAML is staged where the agent can read it, so a literal canary in the criterion would plant that canary outside the reference — and an agent that pasted raw cat output of the task file into its findings would fail with no leak having occurred. The pattern matches the real canary but not its own source text, so the only way to trip it is to have read the reference.

Writes are covered separately: the reference tree is hashed at staging and re-verified before grading, so an agent cannot overwrite it to drive a comparison criterion to 1.0.

What this does not do

The permission window stops an agent that passively reads — ls, cat, grep all fail. Two things it does not stop, both documented in the repo. A deliberate re-chmod: chmod(2) is gated on owner-or-CAP_FOWNER and the container runs as root, which owns the copy, so restoring access from inside works. Waiting the window out: the window spans the agent’s turn, and nothing reaps its child processes at the end of one.

Closing the read half fully means running the agent as a non-root uid that does not own the reference, and holding the window for its whole lifetime. That is tracked, not done.

So the honest summary: the container rungs raise the cost of leakage; they are not a boundary against an agent that is actively looking. The 2.3% above is what actively looking produces, which is exactly why the canary matters more than another rung. Prevention against a determined agent is never quite finished; detection is cheap, and it is total. You are not shipping the sandbox to production; you are making sure a number means what it says.

The claim to check on your own suite

This costs a grep, not a run. Take a suite you already trust, plant a canary in its reference solutions and fixtures, and search the artifacts of the last run. A hit tells you your pass rate includes retrieval — from a record you already had. Nothing tells you which rung you actually need, which is usually lower than the one you were about to build.

Start with your highest pass rates, not your lowest. That is where we found ours, and it is the same lesson as the cross-agent audit from the opposite direction: a result that looks reasonable is not evidence that the machinery under it worked.

Coder Eval is open source (Apache-2.0). The container rungs are documented in Docker isolation, reference solutions in the task definition guide, and the introduction is the shorter road in.

uv tool install coder-eval   # or: pip install coder-eval