GameDevBench

A benchmark suite for evaluating LLM agents on game development tasks.

Paper: GameDevBench: A Comprehensive Benchmark for Game Development

Overview

GameDevBench contains 132 game development tasks to evaluate LLM agents' ability to complete game development problems in the Godot game engine.

Installation

Prerequisites

1. Godot 4.x - Download and install from godotengine.org

   • Ensure godot is available in your PATH, or set GODOT_EXEC_PATH environment variable

2. Python 3.10+ - Required for all agents

   • Python 3.12+ - Required for OpenHands agent

Install Agents

Install the agent(s) you want to use:

• Claude Code - Claude Code
• Codex - Codex
• Gemini CLI - Gemini CLI
• OpenHands - OpenHands

Setup Tasks

Before running the benchmark, unzip the tasks folder:

unzip tasks.zip

Note: The tasks are distributed as a zip file to prevent accidental data leakage.

Configuration

Environment Variables

You can use the built-in plans for claude-code, codex, and gemini-cli, or provide API keys directly. For OpenHands you must provide your own API keys. See .env.example for a complete list of optional environment variables.

Usage

Running the Benchmark

uv run python gamedevbench/src/benchmark_runner.py \
  --agent AGENT \
  --model MODEL \
  run --task-list tasks.yaml

Available Agents

• claude-code - Anthropic's Claude Code CLI
• codex - OpenAI Codex
• gemini-cli - Google Gemini CLI
• openhands - OpenHands (requires Python 3.12+)

Command-Line Options

• --agent AGENT - Agent to use (required)
• --model MODEL - Model name (e.g., claude-sonnet-4.5-20250929)
• --enable-mcp - Enable MCP (Model Context Protocol) server for supported agents
  • Provides screenshot capabilities to the agent
  • Note: MCP server requires macOS (see limitations below)
• --use-runtime-video - Enable runtime video mode
  • Appends Godot runtime instructions to prompts
  • Helps agents understand how to run and test their changes
• --skip-display - Skip tasks that require display
• run --task-list FILE - Run tasks from YAML file (e.g., tasks.yaml)

Platform Limitations

macOS-only Features:

• MCP server screenshot functionality (--enable-mcp) currently only works on macOS
  • Uses AppleScript for display capture
  • Requires setting GODOT_SCREENSHOT_DISPLAY environment variable to correct display number

Results

Benchmark results are saved to results/ directory with the following information:

• Task success/failure status
• Token usage and costs
• Execution time
• Validation results

Citation

@misc{chi2026gamedevbenchevaluatingagenticcapabilities,
      title={GameDevBench: Evaluating Agentic Capabilities Through Game Development},
      author={Wayne Chi and Yixiong Fang and Arnav Yayavaram and Siddharth Yayavaram and Seth Karten and Qiuhong Anna Wei and Runkun Chen and Alexander Wang and Valerie Chen and Ameet Talwalkar and Chris Donahue},
      year={2026},
      eprint={2602.11103},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.11103},
}

License

gamedevbench-bp