Coder Eval

Gate your skill PRs: coding-agent evals in 25 lines of GitHub Actions

· Coder Eval Team · 10 min read

  • ci
  • github-actions
  • junit
  • skills
  • evaluation

Your test suite catches broken functions. Nothing catches a broken agent.

A skill’s description gets a well-meaning copyedit and the model stops selecting it. A dependency bump changes what a CLI prints and the agent starts misreading its output. A new model version interprets your prompt slightly differently. None of these break a unit test, because none of them are code in the classical sense — they’re behavior, and behavior regresses silently. The first person to notice is a user, days later, wondering why the thing that worked last week doesn’t.

Ordinary code earned CI decades ago. Agent behavior deserves the same treatment: a skill that quietly stops triggering should fail a build, not a user.

Coder Eval now ships that as a packaged gate. It’s on the GitHub Actions Marketplace as coder_eval, and here is the whole workflow:

name: Skill evals
on:
  pull_request:
    paths: ["skills/**", "tests/tasks/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    # Fork PRs don't receive secrets — see the security note below.
    if: github.event.pull_request.head.repo.full_name == github.repository
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v4
        with: { node-version: "20" }
      - run: npm install -g @anthropic-ai/claude-code


      - uses: UiPath/coder_eval@v0
        with:
          tasks: tests/tasks/**/*.yaml
          model: claude-sonnet-5
          env: |
            ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
      - uses: mikepenz/action-junit-report@v5
        if: always() && hashFiles('coder-eval-junit.xml') != ''
        with:
          report_paths: coder-eval-junit.xml

The action installs a pinned coder-eval release from PyPI, runs your task YAMLs, writes a JUnit XML report, appends the run summary to the job summary page, and fails the step if any task or suite gate fails. The last step feeds the JUnit file to GitHub’s test-report renderer, so eval results show up as annotations and a test table in the PR — the same UI your unit tests use.

The two Node steps are there because the action is agent-agnostic: it installs the evaluator, not the evaluated. Tasks running on the default claude-code agent need the claude CLI on PATH, and your job supplies it. Swap those two steps for your own agent’s runtime if you’re evaluating Codex or Antigravity.

What lands in the PR

The JUnit mapping is designed to be read by a human skimming a red check, not just counted by a machine.

How a run maps onto a JUnit report The run id is the root testsuites element. Under it, one testsuite per variant — baseline and with-anchor — each holding the same three testcases, named by task with a replicate index. In baseline, both replicates of row-01 pass and row-02 fails; in with-anchor all three pass, and each case carries properties such as weighted_score and total_cost_usd. Two synthetic suites follow: skipped, carrying each skip reason, and suite-gates, where baseline's activation suite fails at recall.yes 0.62 against a floor of 0.7. <testsuites name="2026-08-04_14-22-07"> <testsuite name="baseline"> activation/row-01[00] activation/row-01[01] activation/row-02[00] ok ok FAIL <testsuite name="with-anchor"> activation/row-01[00] activation/row-01[01] activation/row-02[00] ok ok ok properties: weighted_score, total_cost_usd, … <testsuite name="skipped"> tests/tasks/legacy_flow reason: skip: true <testsuite name="suite-gates"> baseline/activation recall.yes: 0.62 < 0.7
The whole tree from a two-variant run — the shape any JUnit ingester sees.

The report is built from the finalized run directory on disk — the run.json spine, any suite.json gates, and per-failed-row task.json for detail — which means the same code path serves both entry points. You can regenerate it from any finished run without re-running anything:

Terminal window
coder-eval run tasks/*.yaml --junit-xml coder-eval-junit.xml   # during the run
coder-eval report runs/latest -f junit                         # after the fact

And because it’s plain JUnit XML, nothing here is GitHub-specific. Azure DevOps ingests the same file via PublishTestResults@2, GitLab via artifacts:reports:junit.

Gating semantics: thresholds, floors, and exit-code honesty

For dataset-backed suites — say, an activation dataset with dozens of prompts that should and shouldn’t trigger a skill — the pass/fail verdict shouldn’t hinge on any single row. LLMs are stochastic; one flaky row failing the build trains everyone to click re-run. Instead, criteria declare suite_thresholds:

success_criteria:
  - type: skill_triggered
    skill_name: my-skill
    expected_skill: "${row.expected_skill}" # "" on rows where it must not fire
    suite_thresholds:
      recall.yes: 0.70 # fired on ≥70% of the rows that needed it
      precision.yes: 0.80 # ≤20% false activations

The suite aggregates per-row results into accuracy, per-label precision/recall/F1, and a confusion matrix, and the gate fails only when an aggregate metric drops below its floor. That’s the regression signal you actually want: “activation recall fell from 0.95 to 0.62” is actionable; “row 17 failed once” is noise. (We wrote about measuring activation this way on a suite of ~1,400 labeled prompts.)

That shape runs in public: each of the 24 skill_triggered criteria in the UiPath skills repo’s activation suite carries suite_thresholds: {recall.yes: 0.70}. Its PR gate is stricter still — activation-gate.yml fires when a PR edits a skill’s SKILL.md frontmatter, re-runs the activation eval over just that skill’s positives, and fails if recall.yes lands more than 10 points below a pinned per-skill baseline. That gate predates the action and drives the CLI from a small Python wrapper, comparing against the baseline there rather than in YAML — but it’s the same bet this post is making: a description edit that costs you activation should turn a check red while it’s still a diff.

One property of the action’s exit handling is worth calling out, because CI tooling routinely gets it wrong: the report is written before the gate evaluates. coder-eval run --junit-xml writes the XML after results are persisted but before the failure exit code is raised, and the action likewise emits its run-dir and junit-path outputs and the job summary before exiting with the captured code. A red run still produces a complete report — which is precisely when you need it most, and what lets downstream steps guarded with if: always() find the artifacts. A gate that only reports on success is a gate you’ll end up debugging blind.

There’s also an optional minimum-task-score input — a strict floor in [0.0, 1.0] that fails the step if any scored task, in any variant, falls below it. It layers on top of coder-eval’s own verdict rather than replacing it, and both surface.

Why a composite action, not a Docker action

The obvious way to package a CLI as an action is a Docker-image action. We deliberately didn’t, because the action is agent-agnostic. A Docker image would have to bake in an agent runtime, and whichever one we picked would be wrong for half the users. Coder Eval evaluates Claude Code, Codex, Google Antigravity (Gemini), and anything registered through its plugin SPI — so the caller brings their own agent CLI and their own credentials.

There’s one escape hatch: version: local installs from the action checkout instead of PyPI, which is how we dogfood it. Every same-repo PR to the coder_eval repo runs the action against its own in-flight code on a cheap smoke task, then asserts the JUnit file is well-formed and the outputs point at a real run directory. If the action breaks, the framework’s own PRs go red first.

Security sidebar (please read this one)

An eval run executes agent-generated code, so treat the workflow like anything else running semi-trusted code. Never expose secrets to fork PRs — above all, don’t pair pull_request_target with a checkout of the PR head, which hands your credentials to arbitrary fork code; use pull_request with the same-repo condition from the workflow above, as coder_eval’s own dogfood job does, and let a nightly run catch a fork’s regression after merge. The default tempdir driver is isolation, not a security boundary: for tasks you don’t control, use the Docker driver. On the action’s own side, inputs reach the shell through env: rather than ${{ }} interpolation inside script bodies, so nothing user-controlled becomes script text, and the passthrough is scoped to the run step — never written to $GITHUB_ENV. The one deliberate exception is extra-args, documented as trusted caller input, so never wire it to anything a PR author can influence. And the JUnit XML is built, never parsed: no XML parser in the production path means no XXE surface to land on.

The money question

Evals cost real money per run — every task is live agent turns against a live model. Control it on two axes.

Scope, so you’re not running everything on every push. Keep PR runs to a smoke subset: a task glob covering only what the PR touches (that’s what the paths: filter and the tasks: input are doing above), a tags filter, or a dataset --sample through extra-args. Keep the full suite — every variant, every dataset row — on a nightly schedule, where a regression still surfaces within a day but doesn’t tax every iteration. A 20-row smoke gate finishes in a handful of minutes, which is the other reason to keep the PR path small: nobody waits half an hour for a check.

Caps, so a runaway task can’t invent a bill. run_limits takes cumulative budget caps alongside the structural ones:

run_limits:
  max_turns: 15
  max_usd: 0.50 # abort this task if it exceeds 50 cents
  max_total_tokens: 200000

A breach aborts the task with a distinct status (COST_BUDGET_EXCEEDED / TOKEN_BUDGET_EXCEEDED) rather than silently continuing, so a prompt change that sends the agent into a loop shows up as a failed task with an obvious cause instead of a surprise invoice. Set these on dataset-backed suites in particular, where one pathological row multiplies by the row count. Cheap-and-always beats thorough-and-never.

What green looks like

Wire this up and the outcome is concrete: if your skill’s activation recall drops below the floor you set, your next PR check goes red — before merge, with the failing metric named in the PR UI and the per-criterion breakdown one click away. No dashboard to remember to check, no user filing the bug for you. And when it’s green, it’s green for a reason you can point at: a number, over a sample, above a floor you chose deliberately.

If you haven’t built the eval suite yet, start with How to test Claude Code skills — then add the twenty-five lines above and let CI keep you honest. The CI gate reference covers every input, and Tutorial 02 walks the whole setup, including a hand-rolled workflow if you’d rather not use the action at all.

Coder Eval is open source: github.com/UiPath/coder_eval · Marketplace · docs · uv tool install coder-eval

← All posts