Introducing Coder Eval: evaluate coding agents on your tasks, not a leaderboard

July 23, 2026 · Coder Eval Team · 7 min read

announcement
agent-evals
claude-code
skills
benchmarking

You changed something. You swapped the model behind your coding agent, rewrote a skill description, or tightened a prompt. The next few sessions seem better.

How do you know?

Right now you have three options, and each one has a gap.

Fixed public benchmarks. SWE-bench and SkillsBench rank models on a canonical dataset. That is useful for choosing a model, but it cannot answer your question — you cannot author tasks for your codebase, your CLI, or your skills. A leaderboard tells you which model is better at someone else’s tasks.

Output-centric eval harnesses. Many of these grade the text that comes back from an API call. But a coding agent is not a string generator. It edits files, runs commands, and makes decisions across a multi-turn trajectory. The thing you usually need to score is the state of a working directory after the agent is done — plus what it did to get there — not a completion.

Hand-rolled scripts. Everyone’s first move, including ours. You can build sandboxing, weighting, cost telemetry, statistics, and CI gates into them — but you have to build all of it, and then keep it alive. In practice they work for a week and then rot.

Coder Eval is the middle ground we wanted: an open-source (Apache-2.0) framework that runs a real agent — Claude Code, Codex, or Google Antigravity (Gemini) — in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced.

uv tool install coder-eval   # or: pip install coder-eval

One YAML file in, one scored, reproducible run directory out.

One complete task, start to finish

A task is a single YAML file: a prompt, an agent config, and success criteria. Here is a real one from the repo (lightly trimmed):

task_id: "hello_date"
description: "Create a Python script that prints a greeting and today's date."
initial_prompt: >
  Create a Python file named app.py in the current working directory that
  prints 'Hello, Claude!' on one line, and today's date in YYYY-MM-DD format
  on the next line. Use the datetime module. Then run the script with:
  python app.py

agent:
  type: "claude-code"
  permission_mode: "acceptEdits"
  allowed_tools: ["Read", "Write", "Bash"]

success_criteria:
  - type: "file_exists"
    path: "app.py"
    description: "The file app.py must be created."
  - type: "file_contains"
    path: "app.py"
    includes: ["Hello, Claude!", "datetime"]
    description: "The script must contain the required string and import."
  - type: "run_command"
    command: "python app.py"
    timeout: 10
    description: "The script must execute successfully."

Three commands take it from file to scored result:

coder-eval plan tasks/hello_date.yaml   # validate the task — no tokens spent
coder-eval run  tasks/hello_date.yaml   # sandbox + real agent + scoring
coder-eval report runs/latest           # read the result

run creates an isolated sandbox (a temp directory with its own venv by default, or a fresh Docker container if you want hard isolation), sends the prompt to the actual agent, and records every tool call it makes. When the agent finishes, the criteria are checked against the final sandbox state: does app.py exist, does it contain the required strings, does it run cleanly?

The run directory holds experiment-level summaries, and each task attempt is stored under runs/<timestamp>/<variant>/<task>/<replicate>/ — a task.json with per-criterion scores, the full transcript, token counts, and cost; a task.log; and an artifacts/ copy of what the agent produced. report renders it as a table: one row per criterion with its score, weight, and pass/fail against its threshold, plus turn counts, tokens, and cost for the run.

A task this size costs a few cents to run — you’re paying for one short agent session, and the report tells you exactly how much.

Scores, not pass/fail

Every criterion returns a continuous score from 0.0 to 1.0, with a weight (default 1.0) and a pass_threshold (default 0.9). file_contains with four required strings and three present scores 0.75, not “failed.” A run isn’t just green or red — it’s a number you can track, compare, and regress against.

What else is in the box

Fourteen criterion types, but four of them are the reason the framework exists:

Trajectory checks. command_executed verifies the agent used (or avoided) specific tools. skill_triggered asks whether the agent actually engaged a target skill — so skill activation stops being a vibe and becomes something you test like code. Classification-style criteria roll up to suite-level precision, recall, and F1, which means a labeled activation dataset gives you a confusion matrix over agent behavior.
agent_judge. For work that only a rubric can grade, the judge gets its own sandbox copy and real tools — Bash, Read, Grep — to investigate with, rather than an opinion about a diff.
Dataset fan-out. One task definition expands over inline rows or a JSONL file with ${row.field} substitution: one YAML file, hundreds of test rows.
Experiments. The experiment layer runs the same tasks across variants — model vs. model, skill on vs. off, prompt vs. prompt — and the comparison report includes bootstrap confidence intervals and a paired mean-difference test. Agents are nondeterministic; a single run of each variant proves nothing, and the reports are built around that fact.

Underneath all of it: every tool call, every token, and the cost of every run are recorded and attributed, including sub-agent usage. When a change makes your suite more expensive, you see it in the same report that shows whether it made the suite better. And the CLI exits non-zero when a suite gate fails, so a GitHub Actions job can fail the build on a regression.

What it is not

Coder Eval is not an agentic-coding benchmark and not a leaderboard. It ships example tasks, not a canonical scored dataset, and it will not tell you whether Claude is “better” than Codex in general. It measures how effective your CLI, skills, and configuration are when used by coding agents — which is a different question, and usually the one you actually have.

Some honest limits: tasks execute real code, and the default tempdir sandbox is not a security boundary — run untrusted tasks under the Docker driver. You bring your own model credentials. Python 3.13+ only.

A falsifiable takeaway

Here is the claim this project stands on: any “it seems better” change to a coding-agent setup — a skill edit, a prompt tweak, a model swap — can be turned into a task suite that either confirms the improvement with a number or shows it was noise. In our experience that suite takes an afternoon to write. If you find a coding-agent behavior you care about that the criterion types can’t capture, that’s a gap in our model of the problem — please file an issue and prove us wrong.

Try it

Quickstart: Tutorial 01 — Your First Evaluation gets you from install to a scored run in about five minutes.
Source: github.com/UiPath/coder_eval — issues and PRs welcome, and if you’re using it, add yourself to ADOPTERS.md.