Running Google Antigravity (Gemini) in Coder Eval

Overview

Coder Eval can run Google’s Antigravity agent — powered by Gemini — as the agent under evaluation. Set agent.type: antigravity in a task and the rest of the framework (sandbox, scoring, telemetry, reports) works unchanged.

Under the hood AntigravityAgent drives the Antigravity SDK’s local harness (the localharness binary bundled in the wheel) rather than the branded agy CLI or the remote Interactions API. The local harness is the only surface that can (a) authenticate headlessly with an API key and (b) make edits land in the sandbox working directory — both required for an unattended eval.

Setup

1. Install the Antigravity SDK

pip install 'coder-eval[antigravity]'

This pulls in google-antigravity (pinned to 0.1.7), whose wheel bundles the platform localharness binary. As with the other agents the SDK is imported lazily — a base install without the extra still runs end-to-end; Antigravity tasks fail at dispatch with a clear hint to install the extra.

2. Authentication

Antigravity authenticates against the Gemini Developer API with a single credential:

GEMINI_API_KEY=<your-gemini-api-key>

Set it in .env (or the environment). That is the only credential Antigravity reads — there is no base-URL, project, or region override for this agent (unlike Codex’s CODEX_BASE_URL / Azure routing). If GEMINI_API_KEY is unset the SDK raises a clear error on first use.

The --backend flag (direct / bedrock) governs Claude routing only; it has no effect on the Antigravity agent.

Usage

Command line

coder-eval run tasks/agents/antigravity_hello_world.yaml --type antigravity

Or override the agent type for every task in an experiment:

coder-eval run experiments/model-comparison.yaml --type antigravity

Task definition (YAML)

agent:
  type: antigravity
  model: gemini-3.1-pro-preview   # optional; see "Model selection" below
  thinking_level: medium          # minimal | low | medium | high (default: medium)
  plugins:
    - type: local
      path: "$SKILLS_PLUGIN_PATH"  # a directory of skills (SKILL.md), env-expanded

run_limits:
  max_turns: 5
  task_timeout: 360
  turn_timeout: 300

success_criteria:
  - type: file_exists
    path: "hello.py"
    description: "hello.py must be created"

Two runnable examples ship in the repo:

• tasks/agents/antigravity_hello_world.yaml — tempdir driver
• tasks/agents/antigravity_hello_world_docker.yaml — docker driver

Model selection

The resolved model is the first of:

1. agent.model in the task YAML
2. ANTIGRAVITY_MODEL in the environment
3. the built-in default, gemini-3.5-flash

Note: the shipped example tasks pin gemini-3.1-pro-preview. Pin agent.model explicitly in any task you care about reproducing — don’t rely on the built-in default, which tracks the current recommended coding model and may change.

thinking_level

Antigravity exposes a thinking_level field (minimal / low / medium / high, default medium) that maps to Gemini’s thinking budget. This is Antigravity-specific — Claude Code and Codex don’t take this field. Thinking tokens are billed as output tokens (see Telemetry).

Skills (SKILL.md)

Antigravity supports Agent Skills (SKILL.md) natively. Skill directories are discovered from:

1. agent.plugins entries with type: local and a path (env vars in the path are expanded at runtime), and
2. the runtime plugin directory passed to start().

For each source, the harness is handed whichever of <source>/skills or <source> directly contains subdirectories with a SKILL.md. Unlike Codex — which symlinks skills into .agents/skills/ — Antigravity is given the search-path roots directly via the SDK’s skills_paths, and those roots are also added to the harness workspaces allowlist (otherwise the agent would find a skill but be denied the SKILL.md read as out-of-workspace). The agent logs a loud warning if a skills path can’t be resolved or if zero skills are discovered.

Permissions & tools — important differences

Antigravity ignores permission_mode, allowed_tools, and disallowed_tools. The local harness runs in a single unconditional mode: every tool call (including run_command) is approved via an allow-all policy, and file tools are restricted to the configured workspaces (the sandbox working directory plus any skill roots).

The trust boundary for an Antigravity run is therefore the sandbox, not the agent config. Run untrusted tasks under the Docker driver; the tempdir driver is not a security boundary. This mirrors the reality that the bypassPermissions-equivalent behavior is always on for this backend.

Those inherited fields still exist on the config for schema uniformity but have no runtime effect here — don’t rely on them to gate Antigravity.

Telemetry

Each communicate() call is one logical turn; the standard TurnRecord is built by the shared EventCollector, so Antigravity runs report the same per-turn structure as every other agent.

• Tool-name normalization. Antigravity’s built-in tool names are mapped to the canonical Claude-style names so cross-agent criteria work: run_command → Bash, create_file → Write, edit_file → Edit, view_file → Read, search_directory → Grep, find_file → Glob, list_directory → LS, start_subagent → Task, search_web → WebSearch. Argument keys are also normalized (e.g. command_line → command) so command_executed criteria key on the same params across agents.
• Tokens. Gemini usage maps to Coder Eval’s four buckets: uncached input, cache read (cached_content_token_count), output, and cache_creation is always 0 (Gemini bills no separate cache-write). Thinking tokens are folded into output. Per-generation usage is cut into AssistantMessages that sum exactly to the turn total, so there is typically no reconciliation residual.
• Commands. Each tool call is captured as CommandTelemetry with status, an untruncated result_summary, error message, and duration. Harness-appended result fields (exit_code, combined_output, diff_block, stdout/stderr, …) are stripped from the recorded parameters so tool output never leaks into the recorded call — which also keeps it from false-positiving a skill_triggered substring match.

Known limitations

1. No cooperative early stop. Antigravity does not support the should_stop seam, so run_limits.stop_early is unsupported for this agent (it errors at resolution, as it does for any non-Claude agent).
2. No endpoint routing. Only GEMINI_API_KEY + ANTIGRAVITY_MODEL are read — there is no base-URL, project, region, or gateway override.
3. Default-model drift. The runtime fallback (gemini-3.5-flash) may differ from what a given release’s docs or example tasks pin; always set agent.model explicitly for reproducible runs.
4. kill_sync() is best-effort. The SDK’s cancel/disconnect are async-only, so the watchdog’s synchronous kill only flips agent state to ERROR; real teardown happens on the subsequent async stop().
5. Process-global spawn lock. The SDK spawns localharness via a subprocess with no env-injection seam, so the agent transiently mutates PATH across the spawn under a process-wide lock. This serializes harness startup across concurrent tasks (it does not serialize the turns themselves).

Running in Docker

The docker driver works the same as for other agents (see Docker Isolation). The localharness binary ships inside the image via the [antigravity] extra, and GEMINI_API_KEY + ANTIGRAVITY_MODEL are on the container env passthrough allowlist, so they are forwarded automatically.

coder-eval run tasks/agents/antigravity_hello_world_docker.yaml

References

• Agent Skills specification
• Codex Agent Guide — the sibling third-party-agent guide
• Claude Code Agent — the default agent
• Extending Coder Eval — how agents register via the plugin SPI