How to test Claude Code skills: from 5 manual prompts to a 200-row eval suite

You wrote a Claude Code skill. It works — you watched it work. (If you haven’t met the framework this post uses, start here.) The question this post answers is: how do you keep knowing it works, next week, on the next model, after the next edit to its description?

The answer is a ladder. Each step is a real, runnable move up from the last, and by the top you have a 200-row evaluation suite with aggregate quality gates. Here is the shape of a single step, so you know where we’re headed:

task_id: "sql-formatter-basic"
description: "Skill triggers and formats the starter SQL file."
initial_prompt: "Format the queries in queries.sql to a consistent style."

success_criteria:
  - type: "skill_triggered"
    skill_name: sql-formatter
    expected_skill: sql-formatter
    description: "The sql-formatter skill must fire."

Everything below builds toward running that — and 199 siblings of it — automatically.

Step 0 — where everyone starts

You open Claude Code, type a prompt that should trigger the skill, and watch. Did the Skill tool fire? Does the output look right? You try four or five phrasings, fix the skill description when one misses, and call it good.

This is genuinely fine for authoring. It’s the fastest feedback loop there is, and interactive skill-testing dev loops — like the eval workflow built into Anthropic’s skill-creator — make it even better: they help you iterate on descriptions and catch obvious misses while the skill is still soft clay.

But be honest about what this is not. It’s an interactive session: nothing is versioned, nothing is re-runnable on demand, nothing can gate a merge, and n=5 supports no statistics. Each attempt also runs in your live working directory, so state from attempt two leaks into attempt three — the skill may “trigger” because a file from the previous try is sitting there. When a model update ships three weeks later and the skill quietly stops firing, your five manual prompts from March tell you nothing. A dev loop is not a test suite.

Step 1 — your first task file

The first real step is capturing one of those manual prompts as a task YAML: a prompt, a sandbox with starter files, and criteria that score whatever the agent leaves behind.

task_id: "sql-formatter-basic"
description: "Skill triggers on a direct formatting request and produces valid output."
initial_prompt: >
  Format the queries in queries.sql to a consistent style,
  writing the result to queries_formatted.sql.

agent:
  type: "claude-code"
  permission_mode: "acceptEdits"
  allowed_tools: ["Skill", "Read", "Write", "Bash"]
  setting_sources: [] # isolate the sandbox from your own CLAUDE.md
  plugins:
    - type: "local"
      path: "${SKILLS_PLUGIN_PATH}" # the plugin root; env vars are expanded

run_limits:
  max_turns: 15

sandbox:
  template_sources:
    - type: "starter_files"
      files:
        - path: "queries.sql"
          content: |
            select id,name from users where active=1;
            SELECT * FROM orders o JOIN users u ON o.user_id=u.id;
        - path: "check_sql.py"
          content: |
            """Stand-in for your real linter: the file must be non-empty,
            the last statement must be terminated, and parens must balance."""
            import pathlib
            import sys

            text = pathlib.Path(sys.argv[1]).read_text()
            statements = [s for s in text.split(";") if s.strip()]
            terminated = text.rstrip().endswith(";")
            ok = bool(statements) and terminated and text.count("(") == text.count(")")
            sys.exit(0 if ok else 1)

success_criteria:
  - type: "skill_triggered"
    skill_name: sql-formatter
    expected_skill: sql-formatter
    description: "The sql-formatter skill must fire."
    weight: 2.0

  - type: "file_exists"
    path: "queries_formatted.sql"
    description: "Formatted output file exists."

  - type: "run_command"
    command: "python check_sql.py queries_formatted.sql"
    timeout: 30
    description: "Output passes the structural check."

Three criteria, three different questions. skill_triggered checks the trajectory — did the agent actually reach for the skill? On Claude Code that’s an explicit Skill tool call; on harnesses without a Skill tool, it’s the agent opening the skill’s files. file_exists checks the artifact. run_command checks the outcome by executing something against it. Each scores 0.0–1.0 and carries a weight (default 1.0) and pass_threshold (default 0.9); the task passes when every gating criterion clears its threshold.

One caveat about shipping the checker as a starter file: it lives in the sandbox, so the agent can read it — and edit it. That’s fine for a tutorial and convenient while iterating, but for a suite that gates merges, keep graders where the agent can’t reach them (a run_command that invokes a tool from the host image, or a file_check on the artifact itself).

Here’s the layout on disk, so it’s clear where everything lives:

my-skills-repo/
├── skills-plugin/                    # a plugin root, not a bare folder of skills
│   ├── .claude-plugin/
│   │   └── plugin.json               # {"name": "sql-skills", "version": "0.1.0"}
│   └── skills/
│       └── sql-formatter/
│           └── SKILL.md              # the skill under test
└── evals/
    ├── tasks/
    │   ├── sql-formatter-basic.yaml
    │   └── sql-formatter-activation.yaml
    └── datasets/
        └── sql_formatter_activation.jsonl

plugins.path must point at that plugin root — the directory holding .claude-plugin/plugin.json, with the skills one level down under skills/. Point it at a bare directory of skill folders and nothing is loaded, silently: the run completes, the agent solves the task by hand, and skill_triggered scores 0. It reads exactly like a skill the model chose not to use.

Validate it without spending tokens, then run:

cd evals                                          # commands below are relative to here
export SKILLS_PLUGIN_PATH="$PWD/../skills-plugin" # the plugin root, absolute

coder-eval plan tasks/sql-formatter-basic.yaml
coder-eval run  tasks/sql-formatter-basic.yaml
coder-eval report runs/latest

Every run happens in a fresh sandbox — no state leaks between attempts — and everything is recorded. Under runs/latest/<variant>/<task>/<NN>/ you get task.json (per-criterion scores, the full transcript, every tool call with duration and status, token usage and cost), task.log, and an artifacts/ copy of what the agent produced. This is the difference in kind from step 0: the run is a file you can diff, not a memory you can misremember.

What it still isn’t: a sample. One prompt tests one phrasing — and even that one prompt can flip between runs. Skills fail on phrasings you didn’t think to type.

Step 2 — from one prompt to a dataset

The fix is not writing 200 task files. A task with a dataset block fans out into one sub-task per row, with ${row.<field>} substituted into initial_prompt and into string fields of the success criteria:

task_id: "sql-formatter-activation"
description: "Does sql-formatter trigger when it should — and stay quiet when it shouldn't?"
initial_prompt: "${row.prompt}"

# Carry the agent and sandbox blocks over from step 1 — the plugin especially.
# A task without an `agent:` block inherits the experiment default, which loads
# no plugin: the skill is never offered and every positive row scores 0.
agent:
  type: "claude-code"
  permission_mode: "acceptEdits"
  allowed_tools: ["Skill", "Read", "Write", "Bash"]
  setting_sources: []
  plugins:
    - type: "local"
      path: "${SKILLS_PLUGIN_PATH}"

dataset:
  paths:
    # resolved relative to THIS file, which lives in evals/tasks/
    - "../datasets/sql_formatter_activation.jsonl"

success_criteria:
  - type: "skill_triggered"
    skill_name: sql-formatter
    expected_skill: "${row.expected_skill}"
    description: "Activation matches the row's label."

Give the sandbox the same queries.sql starter file too. Negative rows like “Explain what this query returns” need something to explain, and a row that fails because the file wasn’t there tells you nothing about activation.

The JSONL is just labeled rows — positives where the skill should fire, and negatives where it must not:

{"id": "direct-01", "prompt": "Format the queries in queries.sql.", "expected_skill": "sql-formatter"}
{"id": "oblique-07", "prompt": "This SQL file is unreadable, clean it up.", "expected_skill": "sql-formatter"}
{"id": "negative-03", "prompt": "Explain what this query returns.", "expected_skill": ""}

Each row must carry the field named by id_field (default id); it becomes a stable suffix on the task id (sql-formatter-activation/oblique-07), so a row’s result is trackable across runs. Inline rows: work too for small sets; paths: scales to hundreds of rows in files you can generate, review, and version.

Writing the 200 rows

The mechanics are the easy part. The rows are the work, and their composition decides what you can learn. A distribution that has held up for us:

Three rules matter more than the exact split:

1. Never copy phrasing from the skill description. If your description says “use for SQL formatting” and your test row says “format my SQL,” you have tested string matching, not generalization. Write rows from real user messages, support tickets, or session logs wherever you can.
2. Label by ownership, not by topic. The question for each row is “which skill should handle this,” and the honest answer is sometimes “none.” Rows you can’t confidently label are a signal that two skills overlap — that’s a finding, not a labeling problem to power through.
3. Review labels with someone who didn’t write the skill. Author-labeled datasets drift toward what the author hoped the skill would do.

Deduplicate before you run: near-identical rows inflate the row count without adding coverage, and they bias the metrics toward whichever phrasing you happened to repeat.

Two hundred agent runs is also two hundred agent runs, so sampling matters:

# Smoke: the same fixed-seed 20 rows every time — cheap and comparable run-to-run
coder-eval run tasks/sql-formatter-activation.yaml --sample 20

# Coverage: N rows per stratum (default stratify_field: expected_skill),
# re-drawn each run — a nightly job broadens coverage over time
coder-eval run tasks/sql-formatter-activation.yaml --sample-per-stratum 10

The asymmetry is deliberate. --sample N uses a fixed seed because a smoke test wants the same rows each run. Stratified sampling is nondeterministic by default because a nightly suite wants to visit different rows over time while never starving a rare stratum; set dataset.sample_seed in the YAML when you need a pinned, reproducible stratified draw.

Step 3 — aggregate quality gates

A single prompt is a weak test even when it passes: agent behavior is stochastic, so one green row is one sample. Two hundred rows are a distribution, and the right question is whether that distribution clears a floor.

Every criterion gets aggregation for free on a dataset-backed task: count, mean, median, std, min, max over the per-row scores. Classification-style criteria — skill_triggered and classification_match — layer real classification metrics on top: accuracy, macro/weighted/micro F1, per-label precision and recall, and a confusion matrix (for skill_triggered the labels are yes/no).

suite_thresholds turns those metrics into a gate:

success_criteria:
  - type: "skill_triggered"
    skill_name: sql-formatter
    expected_skill: "${row.expected_skill}"
    description: "Activation matches the row's label."
    suite_thresholds:
      recall.yes: 0.80 # fires on at least 80% of positives
      precision.yes: 0.90 # over-triggers on at most ~10% of what it fires on
      recall.no: 0.98 # stays quiet on at least 98% of negatives

If any listed metric falls below its minimum, the suite fails and the CLI exits non-zero — which is all a CI system needs.

Picking the numbers. Don’t invent thresholds; measure them. Run several baseline repetitions of the full suite against the configuration you already trust, look at the observed range across those runs, and set the floor below its lower end with an operational margin. More repetitions buy you a better estimate of the spread — a handful gives you a rough range, not a confidence interval, so keep the margin generous.

Be honest about what the gate then means: a failure is evidence worth investigating, not proof of a regression. Some failures will be unlucky draws, which is the price of gating on a stochastic system at all. What the floor buys you is that the investigation happens at all, before merge, instead of after a user notices. Re-baseline deliberately once you improve the skill — raising a floor you’ve earned is how the gate keeps its value.

This is the load-bearing distinction of the whole ladder: a deterministic check is “a test that must pass every time”; agent behavior is “a distribution with a floor.” Skill activation is genuinely stochastic — the same row can flip between runs — so gating on recall.yes >= 0.80 over 200 rows is meaningful where gating on any single row would be noise.

The metrics also name the two distinct ways a skill goes wrong: missed activations (recall.yes) and a skill that butts into conversations it has no business in (precision.yes and recall.no, both measured by your negative rows). You can only see the second failure mode if your dataset has negatives, which is why 90 of the 200 rows in the table above are negative.

Step 4 — on every PR, and every night

Once the suite is a command that exits non-zero on regression, wiring it into CI is plumbing:

name: Skill evals
on:
  pull_request:
    paths: ["skills/**", "evals/**"]

jobs:
  eval:
    # Fork PRs don't get repository secrets — keep the gate on same-repo PRs.
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with: { node-version: "22" }
      - run: npm install -g @anthropic-ai/claude-code
      - uses: UiPath/coder_eval@v0
        with:
          tasks: evals/tasks/**/*.yaml
          extra-args: "--sample 20" # smoke subset on PRs
          env: |
            ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

Use --sample for the PR gate and --sample-per-stratum for a nightly sweep. The action also writes JUnit XML, so results render natively in the PR checks UI alongside your unit tests. A skill that quietly stops triggering then fails a build instead of a user. For the full treatment — JUnit rendering, gating semantics, cost control, and the security posture that matters when you run agent-generated code in CI — see the CI pipeline tutorial.

The other direction to grow is comparison. The suite you just built measures one configuration; the experiment layer runs variants over the same tasks — skill plugin on vs. off, two skill descriptions, two models — and reports the difference with bootstrap confidence intervals and paired tests rather than vibes. It reuses this exact suite — the dataset, the labels, and the skill_triggered criterion all carry over unchanged. The A/B experiments guide covers the variant syntax.

Your next task file

Each step earned something concrete: a sandbox and a record (step 1), a real sample (step 2), an aggregate gate (step 3), automation (step 4). None of it required abandoning the previous step — the step 1 YAML is still inside the step 3 suite.

The test for whether you’ve actually left step 0 is simple: if you can’t re-run your skill test tomorrow and get a comparable number, you don’t have a test — you have a memory of one.

So write one task file today, for the skill you’re least sure about. uv tool install coder-eval, copy the step 1 YAML above, point it at your skill directory, and run coder-eval plan — that costs nothing and tells you whether the task is well-formed. Then run it for real. Twenty labeled rows by Friday beats a perfect plan for two hundred.

Coder Eval is open source: github.com/UiPath/coder_eval · docs · task definition guide · uv tool install coder-eval