Your skill fires too often: what 50 should-NOT-trigger prompts found

In the companion post we measured whether a skill triggers when it should — recall. This post is about the other half, the half almost nobody tests: does your skill trigger when it shouldn’t?

A skill that misses its cue fails quietly, in its own domain, and someone eventually notices and fixes the description. A skill that over-triggers is often more damaging and much harder to detect. It hijacks conversations that were never about it, injects its instructions into an unrelated context, burns tokens loading reference files nobody needed, and — the really damaging part — produces confidently wrong output, because the skill’s playbook is now steering an agent toward the wrong goal. A recall problem degrades one skill. A precision problem degrades every interaction that brushes against the skill’s trigger surface.

(Not always, of course: a missed diagnostic or safety skill on a high-stakes prompt can be far worse than a spurious one. The asymmetry is about detectability — under-triggering gets reported, over-triggering gets absorbed.)

And precision problems are under-measured by default. A positive-only dataset can measure cross-skill interference — skill B firing on a prompt labeled for skill A is a real false positive, and stacked criteria will catch it. What it leaves untested is the much larger region where no skill should trigger at all, which is where most real traffic lives.

Building the negative set

In Coder Eval, an activation eval is one dataset-backed task: each row is a prompt labeled with its true skill, and the task stacks one skill_triggered criterion per skill in the catalog. Every row gets evaluated against every criterion, so each skill gets its own confusion matrix from the same agent traces.

Negative rows are just rows whose expected skill is the empty string:

task_id: "skill-activation"
description: "Single-turn skill activation over labeled prompts"
initial_prompt: "${row.prompt}"

dataset:
  paths: [activation.jsonl]
  # positive row: {"id": "flow-001", "prompt": "Convert checkout to a .flow process",
  #                "expected_skill": "acme-flow"}
  # negative row: {"id": "neg-007",  "prompt": "Set up Acme Insights and connect it
  #                to our warehouse", "expected_skill": ""}

success_criteria:
  - type: "skill_triggered"
    description: "acme-flow activation"
    skill_name: acme-flow
    expected_skill: "${row.expected_skill}"
    suite_thresholds:
      recall.yes: 0.70 # fires on the prompts it owns
      precision.yes: 0.95 # of the times it fired, how often was it right
      recall.no: 0.99 # of the rows it should have ignored, how many did it
  - type: "skill_triggered"
    description: "acme-planner activation"
    skill_name: acme-planner
    expected_skill: "${row.expected_skill}"
    suite_thresholds:
      recall.yes: 0.70
      precision.yes: 0.95
      recall.no: 0.99
  # ... one stacked criterion per skill

The semantics are worth spelling out. Each skill_triggered criterion labels the row yes or no on two axes: observed (did skill_name engage on this row — by explicit tool call, or on harnesses without a Skill tool, by the agent reading its files) and expected ("yes" iff expected_skill == skill_name). On a negative row, expected_skill: "" matches no criterion’s skill_name, so every stacked criterion expects no — and any skill that fires is a false positive counted against that skill.

Gate on recall.no, not just precision.yes. This matters more than it looks. Precision depends on the ratio of positives to negatives in your dataset, which makes it a poor regression signal on a dataset you’re still growing. Hold a skill’s behavior completely fixed — same hit rate on the rows it owns, same misfire rate on the rows it doesn’t — and grow the negative set from 10 rows to 60: precision.yes falls from 0.90 to 0.60 while recall.no sits still at 0.90. The skill did not get worse. The dataset got bigger, and precision moved anyway.

recall.no — the fraction of should-stay-quiet decisions that stayed quiet, i.e. specificity — is a direct measurement of the thing this post is about, and it is stable under changes to dataset composition. Gate both: precision.yes tells you how trustworthy a firing is, recall.no tells you how well the skill minds its own business. One practical note: thresholds are matched against emitted metric names, and a name the aggregate never produced records a null and fails the gate rather than passing silently — so a typo like recall.non shows up as a red check, not a green one.

One aside for the other activation style. If your eval has the agent write its routing decision to a file instead (the classification_match criterion), negatives use a sentinel: the checker maps a missing or empty file to the observed label (none) and unrecognized content to (other), so a negative row is written as expected_label: "(none)" — the row passes exactly when the agent wrote nothing. The sentinels appear as their own classes in the confusion matrix, which means “declined to answer” and “answered something off-catalog” stay visible as distinct failure modes instead of vanishing into a generic fail.

Six kinds of negative, and only one of them is hard

Not all negatives are worth the same. A prompt about a competitor’s product, or about baking, tests almost nothing — the model was never going to route it to your skill. The negatives that earn their place sit close to the boundary:

Weight your set toward the top three rows. The bottom two are cheap to write and mostly confirm what you already believe.

What false positives actually look like

On our internal suite — about 20 skills and roughly 1,000 labeled prompts, of which 50 are out-of-scope negatives — the baseline run had 48 of 50 negatives clean. Both misfires were the same failure class: brand-token collisions.

The negative prompts mentioned adjacent products that share the company brand token but that no skill in the catalog covers. “Set up Acme Process Insights…” fired acme-planner; “Build an Acme Insights dashboard…” fired acme-platform. The model saw the brand name, pattern-matched to the nearest sibling skill, and committed. Nothing else in the 50 negatives fired: not the generic coding prompts, not the off-topic ones. The entire false-positive surface was sibling products.

That’s the taxonomy lesson, and it generalizes: if your company ships a product family, your negative set must include sibling-product prompts, because that boundary — inside the brand, outside the skill — is where false positives live.

After a description fix that scoped the affected skills explicitly, negatives went from 48/50 to 50/50. Across the whole stacked matrix, the false-positive rate was 0.14% — 26 hits out of roughly 19,000 negative-side criterion evaluations. Two caveats on that denominator, both worth stating plainly. It counts every should-not-fire check, including the cross-skill checks on positive rows, so it is a much larger and easier population than the 50-row negative set. And those ~19,000 evaluations are not 19,000 independent trials: twenty of them share a single agent trajectory, so they correlate. The rate is a useful order of magnitude, not a confidence interval.

The trade-off nobody talks about

The same experiment that fixed the negatives was primarily a recall intervention: adding explicit “always invoke for X” anchor sentences to skill descriptions. Micro-recall went up 21 points, precision stayed flat at ~0.96, and the negatives improved. The recall lift cost effectively nothing on the false-positive side.

But that’s only true because of what the anchors were: unique tokens — file extensions the skill owns, config filenames only its product emits. A unique token can’t collide, so it buys recall for free. The temptation, once you see the technique work, is to anchor everything: give the struggling review skill “always invoke for ‘review’, ‘audit’, ‘is this good’”. Those are generic verbs, and anchoring on them doesn’t create recall — it transfers it, stealing triggers from the domain skills that currently win those prompts. That’s a routing decision about which skill should own an ambiguous prompt, and it deserves a deliberate call with the eval numbers in front of you, not a quick description patch.

Some skills can’t be anchored — and shouldn’t be

Our generic review skill was the worst performer in the suite, and the anchor treatment didn’t help — recall actually ticked down. It has no unique token: “review my code” prompts route to whichever domain skill matches the artifact. And when someone says “review my XAML”, the domain skill firing is the correct routing — it knows the artifact. The eval was partly wrong, not the skill: it expected the review skill to win prompts that carry no review-specific signal. Sometimes the fix is accepting low recall for a skill that’s structurally a fallback.

And sometimes the fix is structural. Two overlapping sibling skills in our suite were merged into one, and their combined recall went from 0.68 to 0.84. No description got smarter — the competition between siblings was itself a measurable recall tax, and removing the boundary refunded it. If two of your skills keep stealing each other’s prompts, the dataset is telling you they’re one skill.

Write the fifty rows

If your activation eval has no negative rows, your behavior against out-of-scope traffic is unmeasured — and unmeasured precision is usually overestimated. Our own suite looked clean until the negatives said otherwise, and the two failures were in a class no positive-only dataset would ever surface.

So here’s the exercise, and it’s an afternoon’s work. List every product your company ships that sits adjacent to the one your skills cover — the ones that share a brand token but that no skill in your catalog owns. Write ten realistic prompts for each. Label them expected_skill: "", add them to your dataset, and gate on recall.no. If the number that comes back is 50/50, you’ve spent an afternoon buying certainty. If it isn’t, you’ve found the bug before your users did — and you now know exactly which sibling is stealing from which.

Coder Eval is open source: github.com/UiPath/coder_eval · docs · uv tool install coder-eval