Running Claude Code in Coder Eval

Overview

Claude Code is the default agent in Coder Eval — agent.type: claude-code. It ships in the base install (no extra needed; the claude-agent-sdk is a core dependency), and most tasks and tutorials assume it. This guide is the reference for its config surface, authentication, and telemetry; for the agent-agnostic task schema see the Task Definition Guide.

Because agent.type defaults to claude-code in experiments/default.yaml, you can omit the type field entirely.

Setup

Nothing beyond the base install:

uv tool install coder-eval        # or: pip install coder-eval

Provide model credentials (see Authentication) and you’re ready:

coder-eval run tasks/hello_date.yaml     # claude-code is the default

Authentication

Claude Code routing follows the --backend flag (or API_BACKEND env var):

Direct (default)

--backend direct calls the Anthropic API. The SDK picks its own credential, in order:

1. ANTHROPIC_API_KEY in the environment / .env, or
2. a cached claude login (subscription) session.

No key is validated at startup — the SDK fails with a clear error if none is found. The llm_judge / agent_judge criteria also need ANTHROPIC_API_KEY under the direct backend (they call api.anthropic.com).

AWS Bedrock

--backend bedrock routes through Bedrock with bearer-token auth. Set in .env:

Variable	Purpose
AWS_BEARER_TOKEN_BEDROCK	Bedrock bearer token (required)
AWS_REGION	Bedrock region, e.g. eu-north-1 (required)
BEDROCK_MODEL	Cross-region model id, e.g. eu.anthropic.claude-sonnet-5 (required)
BEDROCK_SMALL_MODEL	Small/fast model id (falls back to the main model)

The agent sets CLAUDE_CODE_USE_BEDROCK=1 and forwards these into the SDK subprocess. BEDROCK_MODEL is the route-level default; an explicit --model / -D agent.model=… / task agent.model always wins over it, and bare aliases are auto-qualified to a regional inference profile (eu. / us. / apac. / global.).

For official benchmarking use the direct API — the SDK reports accurate token/cost there. See User Guide → API Routing.

Agent config surface

All fields live under agent: in a task (or an experiment variant). Only type is required; everything else has a default.

agent:
  type: claude-code
  model: claude-sonnet-5                 # optional; omit to use the route default
  permission_mode: acceptEdits           # default | acceptEdits | plan | bypassPermissions
  allowed_tools: ["Read", "Write", "Bash"]
  disallowed_tools: ["WebSearch"]
  setting_sources: []                     # [] = fully isolated (recommended, see below)
  plugins:
    - type: local
      path: "$SKILLS_PLUGIN_PATH"
  system_prompt: "You are a careful engineer."   # or system_prompt_file
  claude_settings:
    permissions:
      deny: ["Read(/etc/**)"]
  sdk_options:
    effort: high

Field	Type / default	Meaning
type	"claude-code"	Agent kind (the default).
model	str | null	Specific model id. Omit to use the backend/route default.
permission_mode	default acceptEdits	default / acceptEdits / plan / bypassPermissions — semantics come from the Claude Code SDK. plan is read-only; bypassPermissions grants full autonomy.
allowed_tools	list[str] | null	Tool allowlist. Unset ⇒ all tools allowed.
disallowed_tools	list[str] | null	Tool denylist. (ToolSearch is always appended for Bedrock parity.)
plugins	list[{type: local, path}]	Local plugin roots; $VAR in path is expanded and resolved to an absolute path. path must hold a skills/ subdirectory — a bare directory of skill directories loads nothing here, though Codex and Antigravity accept it. See Harness parity.
system_prompt	str | null	Appended to the default Claude Code system prompt (via the SDK’s claude_code preset) — the default’s behavioral guidance is kept unless system_prompt_mode: replace opts out. Mutually exclusive with system_prompt_file. An empty or whitespace-only value is treated as unset.
system_prompt_mode	"append" (default) / "replace"	replace sends system_prompt as the entire system prompt (no preset) and requires a non-blank system_prompt / system_prompt_file (validated at load). Used by judge sub-agents and the user simulator, which must not carry the coding-agent persona; rarely needed in tasks — see the migration note.
system_prompt_file	str | null	Path (relative to the task YAML) loaded into system_prompt at resolution. Works with either system_prompt_mode.
setting_sources	list["user"|"project"|"local"] | null	Which host setting sources the SDK reads. Default resolves to ["project"]. See Sandbox isolation.
claude_settings	str | dict | null	Passed to the SDK --settings. A dict is JSON-serialized; a str is a settings file path. Use permissions.deny to block tools/paths.
sdk_options	dict (default {})	Pass-through for ClaudeAgentOptions fields Coder Eval doesn’t own (e.g. effort). Validated at load — an unknown or framework-owned key is a hard error.
ignore_patterns	list[str] | null	Gitignore-style overrides for the workspace copy used by judge sub-agents (supports ! negation).

sdk_options is a deliberate escape hatch. Framework-owned keys (model, permission_mode, allowed_tools, mcp_servers, resume, max_turns, setting_sources, include_partial_messages, …) are rejected there — set those through their typed fields or -D run_limits.*. MCP servers are not a YAML field.

System-prompt reproducibility. In append mode the preset’s dynamic sections (working directory, git status, auto-memory) are excluded so the system prompt stays identical across runs — the per-run sandbox tempdir path would otherwise be baked into it, breaking prompt caching and run comparability. The SDK re-injects the stripped content into the first user message, so the agent loses nothing. Note the default-prompt baseline tracks the installed Claude Code CLI version; environment_info.claude_code_cli in run.json records which version a run used, and environment_info.system_prompt_semantics (append / replace) records the prompt regime — runs predating that marker used replace-on-set / empty-on-unset semantics and are not score-comparable.

Migrating tasks that set system_prompt

system_prompt used to replace Claude Code’s default system prompt. It now appends to it. If your task or experiment sets system_prompt, the agent gains back every default behavioral instruction it was previously running without — parallel tool-call batching, conciseness rules, the Read/Grep/Glob tool preferences, and the default security guardrails.

That is a genuine behavior change, so scores are not comparable across this boundary. Pick one:

• Keep the repair (recommended). Do nothing. Re-baseline any threshold or reference score the task gates on, and expect turn counts to drop on tasks that depend on batched tool calls.
• Preserve the old behavior. Add system_prompt_mode: replace to the agent: block. The configured prompt again becomes the entire system prompt. Only do this if the task intends to run without the default guidance — a prompt that merely adds sandbox policy or a persona does not.

Segment dashboards on environment_info.system_prompt_semantics to keep the two regimes in separate cohorts. Note the append-mode baseline also tracks the installed CLI version, so a CLAUDE_CODE_VERSION bump becomes a score-affecting change (attributable via environment_info.claude_code_cli).

Consumers reading sdk_options.system_prompt. The persisted value changed shape: a plain str in the old regime, a SystemPromptPreset dict ({type, preset, exclude_dynamic_sections, append}) in append mode. Code that string-handles that field needs to branch on the type — see Report schema.

Setting fields from the CLI

Any of these merge-resolve through -D / --set (see User Guide → CLI overrides):

coder-eval run tasks/hello_date.yaml \
  -D agent.model=claude-opus-5 \
  -D agent.permission_mode=plan \
  -D agent.sdk_options.effort=high

Sandbox isolation

setting_sources controls whether the host project’s settings leak into the run. The default (["project"]) lets the SDK discover .mcp.json — but it also pulls in the host project’s CLAUDE.md, settings, and hooks, which for this repo is 20 KB+ injected into every API call (inflating cache-creation tokens and cost).

For tasks that don’t need host MCP servers, set setting_sources: [] for full isolation from the host CLAUDE.md/settings/hooks. (Judge sub-agents and the user simulator force [] for the same reason.)

Skills & plugins

• Skills are engaged via Claude’s explicit Skill tool call; the skill_triggered criterion detects that call and scores skill-activation suites.
• Plugins are supplied as plugins: [{type: local, path: …}]; the path is env-expanded and resolved to an absolute path before being handed to the SDK. It must name a plugin root — a directory holding skills/, so the skill resolves at <path>/skills/<name>/SKILL.md. One level deeper loads nothing, with no error: every positive row of an activation suite then scores 0 and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Codex and Antigravity scan both depths, so this is claude-code-only — see Harness parity.
• A plugin root loads everything the plugin declares, not only skills/: an agents/, commands/ or hooks/ directory beside it becomes visible to the evaluated agent too. Point at a minimal root when the suite must measure one skill in isolation.
• PLUGIN_TOOLS_DIR pins the canonical node_modules/@uipath directory for UiPath CLI plugin discovery; when unset the sandbox derives it from the resolved uip binary. See User Guide → Environment Variables.

Early stop

Claude Code supports the cooperative early-stop seam, as do the Codex and Antigravity agents. When a criterion carries a stop_early: block, a single-shot run ends cleanly at the next tool-call boundary once its armed criteria (those carrying a stop_early: block) are decided — so a raised max_turns isn’t wasted on a smoke run. Early stop errors at resolution for any agent that does not declare supports_cooperative_stop. See the Task Definition Guide for the full contract.

Telemetry

Claude Code produces the richest telemetry of the agents:

• Authoritative billing comes from the SDK’s cumulative model_usage on the terminal result message and reconciles to total_cost_usd — this already includes sub-agent consumption that the per-message stream under-reports.
• Sub-agent accounting is derived by grouping parent_tool_use_id-tagged assistant messages; there is no separate per-sub-agent field. The terminal sub-agent generation (delivered as the Agent tool result, never streamed) is synthesized into one message.
• Reconciliation message. Because the per-message stream slightly under-reports the turn total (a fixed ~512-token input slice is billed on no streamed message, and sub-agent input/cache only partially bubbles up), each turn carries one synthetic role="reconciliation" entry holding the per-bucket residual. The invariant: summing the token buckets across TurnRecord.messages equals the turn total exactly. It carries no cost and is excluded from turn/generation counts.

Set the CODER_EVAL_RAW_SDK_LOG environment variable to 1 to dump every raw SDK event to the task log.

Lifecycle & robustness

• Session resume across turns via the SDK resume option; the session id only advances on clean (non-error) turns.
• Timeouts are enforced by a ThreadedWatchdog (an OS-thread timer immune to event-loop stalls) plus an in-loop wall-clock guard; on breach it SIGKILLs the CLI subprocess and raises TurnTimeoutError with a partial TurnRecord preserved.
• Crashes raise AgentCrashError with assembled stderr; the orchestrator drains the partial turn and rolls back. Cost is backfilled from the rate card when a run is killed before a terminal result message arrives.

References

• Task Definition Guide — the full task/criterion schema
• User Guide — CLI, backends, environment variables
• Codex Agent Guide · Antigravity (Gemini) Agent Guide
• Extending Coder Eval — how agents register via the plugin SPI