Skip to content
Coder Eval

Tutorial 04 — Writing a Task

By the end you’ll have authored a task YAML from scratch — a prompt, an isolated sandbox, a soft turn budget, and three success criteria — validated it without spending tokens, run it, and read the score. ~10 minutes.

Prerequisites

Section titled “Prerequisites”
  • A working coder-eval checkout with an API key configured — see Tutorial 01.

1. Create the task file

Section titled “1. Create the task file”

A task is a single YAML file: the prompt sent to the agent, the agent configuration, and the criteria that score whatever the agent leaves behind in the sandbox. Create tasks/fizzbuzz.yaml:

task_id: "fizzbuzz"
description: "Write and run a classic FizzBuzz script."
initial_prompt: >
  Create a Python file named fizzbuzz.py in the current working directory that
  prints the numbers 1 to 15, one per line, replacing multiples of 3 with
  'Fizz', multiples of 5 with 'Buzz', and multiples of both with 'FizzBuzz'.
  Then run it with: python fizzbuzz.py


agent:
  type: "claude-code"
  model: "claude-sonnet-4-6"
  permission_mode: "acceptEdits"
  setting_sources: []   # isolate the sandbox from your own CLAUDE.md/settings


run_limits:
  expected_turns: 5     # soft target — warns when exceeded, never aborts


success_criteria:
  - type: "file_exists"
    path: "fizzbuzz.py"
    description: "The script must be created."
  - type: "file_contains"
    path: "fizzbuzz.py"
    includes: ["FizzBuzz"]
    description: "The script must produce the combined FizzBuzz case."
  - type: "run_command"
    command: "python fizzbuzz.py"
    timeout: 10
    description: "The script must run successfully."

What each block does:

  • initial_prompt — delivered to the agent inside a fresh sandbox directory; all file paths in the criteria are relative to that sandbox.
  • agent.setting_sources: [] — keeps your own CLAUDE.md and local settings out of the sandbox. Without it, a large host CLAUDE.md is injected into every API call, inflating cache-creation tokens and cost. Leave it out only when the task needs your MCP servers.
  • run_limits.expected_turns — an efficiency target, not a cap: exceeding it logs a warning and adds a report badge but never aborts (use run_limits.max_turns for a hard cap).
  • success_criteria — each criterion scores 0.0–1.0 and supports weight (default 1.0) and pass_threshold (default 0.9). run_command here only checks the exit code; it can also match stdout (expected_stdout) or read a continuous score from stdout — see the Task Definition Guide for all 14 criterion types and every field.

2. Validate without spending tokens

Section titled “2. Validate without spending tokens”
Terminal window
uv run coder-eval plan tasks/fizzbuzz.yaml

plan checks the YAML schema, required tools, and API keys without calling the model. Typos in criterion fields fail here (task models reject unknown keys) instead of after a paid run.

3. Run it

Section titled “3. Run it”
Terminal window
uv run coder-eval run tasks/fizzbuzz.yaml

4. Read the result

Section titled “4. Read the result”
Terminal window
uv run coder-eval report runs/latest

The report shows one row per criterion — score, pass/fail against its threshold, and the weighted total. The agent’s full transcript and the files it created are preserved under runs/latest/<variant>/<task>/<NN>/ (task.json, task.log, artifacts/).

Where to go deeper

Section titled “Where to go deeper”