Does your Claude Code skill actually trigger? Precision and recall over 1,000 prompts

You wrote a Claude Code skill. You typed five prompts at it, it triggered on four, and you shipped it. Then a user reports that the skill never fires for them, or — worse — that it fires on prompts that have nothing to do with it, hijacking work a different skill should own.

Both failure modes are real, both are silent, and neither is caught by typing five prompts. Skill activation is a routing decision: the model reads your skill description alongside every other skill’s description and picks, and that decision shifts when you edit a description, add a sibling skill, or merge two of them. On our internal suite — a catalog of about 20 product-specific authoring and diagnostics skills, with roughly 1,000 labeled prompts — adding one sentence to each skill description moved micro-recall from 46.3% to 67.3%, and took the number of skills clearing 70% recall from 4 to 11. Movement that large under a routine edit cannot be checked by hand once and then trusted.

The fix is to treat activation as what it is — a classification problem — and measure it like one: a labeled dataset, per-skill precision/recall/F1, a confusion view, and thresholds that fail the build.

This post is the case study. If you haven’t built an activation suite yet, start with the tutorial — it covers the dataset design and the task mechanics from scratch, and you don’t need 1,000 rows to begin. Fifty will show you something.

Activation is classification

For each skill and each prompt, exactly one of four things happens:

• True positive — the prompt belongs to the skill, and the skill fired.
• False negative — the prompt belongs to the skill, and it didn’t fire. The silent failure users experience as “your feature doesn’t work.”
• False positive — the prompt does not belong to the skill, and it fired anyway. Either it stole the prompt from the skill that owns it, or it activated on something no skill should touch.
• True negative — out of scope, stayed quiet.

From those you get the standard metrics, per skill: recall (of the prompts this skill owns, what fraction did it catch?) and precision (of the times it fired, how often was it right?). Recall measures the under-trigger problem; precision measures the over-trigger problem. You need both, and that means the dataset needs both polarities: realistic positives per skill, phrased the way users phrase them rather than the way your skill description does, plus a should-NOT-trigger set of prompts that are plausibly in your product’s domain but belong to no skill. Without negatives you cannot distinguish a well-calibrated skill from one that fires on everything — a skill that triggers on 100% of all prompts has perfect recall.

Here is the suite as one task:

task_id: "skill-activation"
description: "Does the right skill trigger for each prompt - and stay quiet otherwise?"
tags: [activation, classification]

agent:
  type: "claude-code"
  permission_mode: "acceptEdits"
  allowed_tools: ["Skill", "Read", "Glob", "Grep"]
  setting_sources: [] # isolate the sandbox from your host CLAUDE.md
  plugins:
    - type: "local"
      path: "${SKILLS_PLUGIN_PATH}" # absolute; env vars are expanded

run_limits:
  max_turns: 1 # one-shot: measure the routing decision, not recovery

dataset:
  paths: # one file per skill, plus the shared negatives
    - uipath-troubleshoot.jsonl
    - uipath-maestro-bpmn.jsonl
    # ... one per skill
    - negative.jsonl
  sample_per_stratum: 10 # optional: N rows per skill per run
  stratify_field: "expected_skill" # the default, shown for clarity

initial_prompt: "${row.prompt}"

success_criteria:
  - type: "skill_triggered"
    description: "uipath-troubleshoot activation"
    skill_name: uipath-troubleshoot
    expected_skill: "${row.expected_skill}"
    suite_thresholds: { recall.yes: 0.70 }
    stop_when: auto
  - type: "skill_triggered"
    description: "uipath-maestro-bpmn activation"
    skill_name: uipath-maestro-bpmn
    expected_skill: "${row.expected_skill}"
    suite_thresholds: { recall.yes: 0.70 }
    stop_when: auto
  # ... one stacked criterion per skill

Each dataset row is a labeled example. These are verbatim rows from the suite this post is about, which is public:

{"id": "uipath-troubleshoot-001", "prompt": "Why did my job fail?", "expected_skill": "uipath-troubleshoot"}
{"id": "uipath-maestro-bpmn-003", "prompt": "Edit my .bpmn file to add a parallel gateway between the validation step and the two downstream service tasks", "expected_skill": "uipath-maestro-bpmn"}
{"id": "negative-016", "prompt": "I need to design a BPMN diagram for our process", "expected_skill": ""}
{"id": "negative-018", "prompt": "Create a connector in Power Automate", "expected_skill": ""}

Look at the middle two. One asks to edit a .bpmn file in a project the BPMN skill owns; the other asks to design a BPMN diagram, in general, for a process that has nothing to do with us. Same domain word, opposite labels — and a skill that can’t tell them apart is exactly the over-triggering failure you’re trying to catch. The prompts that discriminate hardest are the ones a topic-based labeler would get wrong.

Note what is and isn’t gated. Each criterion carries recall.yes only, because the class balance makes it the metric that can move: roughly 50 rows in the set belong to any given skill and the other ~950 don’t, which leaves recall.no trivially high and easy to pass. Precision is still computed and still reported — it just isn’t the number a build should hinge on here. The run exits non-zero if any gate fails, so this drops straight into CI.

One stacked criterion per skill is the part that scales awkwardly by hand — twenty skills is a hundred lines of near-identical YAML. Generate that block from your skill catalog in whatever templating you already use, or keep the shared threshold body in a YAML anchor and vary only skill_name and description. It’s boilerplate, not complexity, but don’t hand-maintain it.

What stacking buys you

Every row carries one criterion per skill, all evaluated against the same single agent run. A row labeled uipath-troubleshoot where the BPMN skill fired instead scores a false negative on the troubleshoot criterion and a false positive on the BPMN one — which is exactly what a per-skill confusion matrix should show you. On our suite that is 20 criteria × ~1,000 rows ≈ 20,000 criterion evaluations from ~1,000 agent runs. The criteria are trace scans rather than extra model calls, so the marginal cost of stacking is nil: you pay for the agent runs, not for the width of the matrix.

Because skill_triggered returns a classification result, the suite aggregator computes accuracy, macro/micro F1, per-label precision.<label> / recall.<label> / f1.<label> (labels are yes/no), and the confusion matrix across all rows — per criterion, so per skill. suite_thresholds gates on any of those metrics, and a failed gate is a non-zero exit code.

What this found on a real suite

Experiment configuration
------------------------
Framework       Coder Eval (see repo releases for the exact pin)
Agent           Claude Code SDK, single-shot, max_turns: 1
Model           one mid-tier frontier model
Dataset         after:  1,001 prompts — 951 positives across 20 skills, 50 negatives
                before:   952 prompts across 19 skills
Criteria        20 stacked skill_triggered checks per row (20,020 evaluations)
Replicates      1 run per prompt
Sampling        none — the full dataset
Parallelism     4 concurrent tasks
Wall clock      ~40 minutes
Variable        skill descriptions, a governance merge, and two new skills

The baseline run was sobering: micro-recall 46.3%. Half the prompts that should have activated a skill activated nothing.

The intervention was one sentence per skill description: an explicit “Always invoke for …” clause naming concrete tokens the skill uniquely owns — file extensions, well-known filenames, artifact types that appear in real prompts and belong to exactly one skill. The result:

A 21-point micro-recall lift, and the precision cost was effectively zero — the false-positive rate after the change was 0.14% — 26 misfires across the 19,069 criterion evaluations where the skill in question was not the one supposed to fire.

The traces also named the dominant failure mode, which was not the one we expected: on most failing rows the agent spent its single turn without invoking any skill. The prompts weren’t being routed to the wrong skill so much as failing to produce a strong enough match for the model to commit at all — which is exactly what an anchor token fixes.

The distribution underneath is the more instructive part. Skills with a unique anchor available moved the most: a diagnostics skill went from 0.16 to 0.68 recall (+0.52), and several others gained +0.35 to +0.40. A feedback skill already sitting at 0.923 had no room to gain and no anchor worth adding, and it read 0.923 after the change too — the closest thing the run had to a control.

And one skill got slightly worse: a cross-cutting review skill — the one that audits structure and quality across every artifact type — went from 0.20 to 0.18, because it has no unique token to anchor on. Only 9 of its 50 prompts ever triggered it. “Review my code,” “audit this project,” “is this good?” route to whichever domain skill matches the artifact being reviewed, which is arguably correct: the flow-authoring skill should answer “review my flow.” That is a finding, not a failure of the method. Some skills are structurally un-anchorable, and forcing an anchor onto this one (“always invoke for ‘review’, ‘audit’…”) would steal triggers from the domain skills that deserve those prompts — a deliberate trade-off, not a quick fix. Part of the gap is also the dataset’s: it expected the review skill to win prompts that carry no review verb at all. Without per-skill numbers you would have applied the same fix everywhere and shipped a regression.

Why this must be a regression suite

Descriptions are only one of the inputs. Activation is a routing decision taken across your whole catalog, so a skill’s numbers move when its neighbors move — and the same pair of runs shows that twice over, in the two catalog changes that rode along with the description edit.

Two sibling governance skills were merged into one. Combined recall went from 0.677 to 0.840, and the credit doesn’t belong to the anchor sentence: it’s that two siblings had been competing for the same prompts and now there is one boundary instead of two, on a prompt set curated to match. Merging skills is not usually thought of as an activation fix. It is one.

The two brand-new skills that joined in the same run are the other half of the argument. Both landed at 0.58 recall, under-firing on plain-English prompts that never mention the tokens their descriptions lean on — one wants a .bpmn file named, the other a piece of product jargon. Neither had a baseline to regress from. The suite is simply how anyone found out they shipped below the bar on day one.

That is the durable lesson. Activation behavior is a function of your descriptions and your skill set — siblings compete — on top of a model and a routing layer that you don’t control at all. A one-off manual check, or even a one-off dataset run, tells you where activation stood on one day under one catalog. Every skill you add re-opens the question for the skills already there, which is why this belongs in a suite you re-run — nightly, or as a PR gate with suite_thresholds — rather than in a spreadsheet from the last time somebody thought to check.

Try this on your own skills

If skill activation were stable, measuring it once would be enough. Here’s the test: label 50 prompts per skill plus a should-not-trigger set, run the suite above before and after your next skill-description edit, added skill, or SDK bump, and compare per-skill recall and precision. If nothing moves more than noise, this post is wrong and you can go back to spot checks. On our suite everything that mattered moved — +21 points of micro-recall from one added sentence, +0.52 on the single skill that gained the most, +0.16 from merging two competing siblings, two new skills shipping at 0.58, and one skill quietly going backwards — and none of it was visible until a labeled dataset turned it into numbers.

And the negative set is where the surprises live. Both misfires in the baseline run were the same shape — two rows that name a product in our own family that no skill in the catalog covers:

{"id": "negative-031", "prompt": "Set up UiPath Process Mining to analyze bottlenecks in my procure-to-pay process", "expected_skill": ""}
{"id": "negative-046", "prompt": "Build a UiPath Insights dashboard to visualize bot performance over the last quarter", "expected_skill": ""}

A skill claimed each of them on the strength of the brand token alone. Any company shipping a product family has that failure class sitting in its catalog, and you only ever see it if your dataset contains prompts that no skill should own.

Coder Eval is open source: github.com/UiPath/coder_eval · docs · uv tool install coder-eval