ARC-AGI-3 World Model Lab

中文说明 | English

A research workbench for reconstructing ARC-AGI-3 games as explicit, executable world models. The project combines three layers:

1. a deterministic C11 actor/rule engine exposed through Python and MCP;
2. a lightweight Grid-SLAM and replay workbench for stable maps, masks, and trajectories;
3. a visual discovery harness for human gold masks, traditional tracking, and multimodal-model experiments over complete action animations.

This is an independent research prototype, not an official ARC Prize project or submission. It deliberately focuses on discovering and verifying game dynamics, not planning or leaderboard optimization.

Why this project exists

ARC-AGI-3 interactions are short, discrete, and visually exact. A useful agent should not have to retain a long stream of screenshots in hidden neural state. Instead, it should be able to construct an external model:

action animation
  -> stable coordinates and candidate masks
  -> action-conditioned object events
  -> actor-owned event-condition-effect rules
  -> deterministic simulation and observation comparison

The engine is both a simulator and a constrained visual-description language. An agent can propose actors and rules, replay observations, inspect residuals, and revise the model without writing arbitrary native code.

Current status

Area	Status	What works now
C world engine	Usable prototype	Actors, static map, ARC colors 0-15, collision, topology transforms, state/model hashes, snapshots, traces, rendering, diffs
ECE rule system	Usable prototype	Action/click/tick/collision/all-event triggers, validated condition expressions, multi-frame effect programs
MCP interface	Usable prototype	Typed actor, topology, rule, snapshot, simulation, render, and observation-comparison operations
Grid-SLAM	Experimental	Integer camera registration, typed grid-corner features, static/dynamic separation, status-bar filtering, object tracks
Web workbench	Integrated	Real/demo backend, full animation playback, replay, map/object views, actor/rule editor, native debug trace
Visual harness	Operational experiment	Full-sequence annotator, unified ROI, traditional bidirectional tracker, UniPixel/K3/Seed adapters and benchmarks
Automatic ontology discovery	Open research	Candidate masks exist, but deciding actor boundaries and grouping still needs model/human hypotheses
Full game reconstruction verifier	Partial	Engine can compare predictions with observations; automatic synthesis of complete game rules is not finished
Planner	Out of scope	No gameplay planner is included yet

World engine

world_engine/ contains the deterministic C11 kernel and Python bindings. The static map is separate from dynamic actors. Every actor can own event-condition-effect (ECE) rules.

Condition code is parsed with a restricted AST and compiled to a native postfix program; Python eval is not used:

actor.visible == true and
(actor.color == 8 or (action.id == 6 and action.x >= actor.x))

The validator checks actor references, property fields, booleans, ARC color ranges, viewport coordinates, and action fields before changing the model.

Effect code uses one validated operation per line and can expose intermediate frames:

frame 0: actor.recolor(8)
frame 1: actor.move(1, 0, block)
frame 2: actor.rotate_to(actor_12, 0, -1)

rotate_to lets an agent use another actor as an explicit post-transform pixel topology. This supports ARC-style rotations and reflections without making the LLM manually rewrite the full frame.

MCP

Start the stdio MCP server:

arc-world-engine-mcp

Set ARC_WORLD_MCP_TRANSPORT=streamable-http for an HTTP transport. The MCP surface intentionally exposes typed cognitive operations rather than arbitrary pixel writes.

Grid-SLAM workbench

The workbench processes every intermediate frame returned by one ARC action. Its current pipeline is:

1. detect typed cell-corner features and estimate an integer camera offset;
2. align frames before computing added/removed masks;
3. update a confidence-weighted static map while excluding dynamic cells;
4. match masks under translation to maintain object trajectories;
5. detect one-cell-wide, two-color monotonic status bars and remove them from ordinary object candidates and map landmarks.

When a level finishes, map and object tracking reset. Exact previous-level object templates remain available as cross-level priors. A full game reset also clears replay, action history, and Grid-SLAM state.

Run the demo

Requirements: Python 3.12+, a C compiler, Node.js 22+, and uv.

make -C world_engine test build
uv venv
uv pip install -e ".[dev]"

cd frontend
npm ci
npm run build
cd ..

python -m grid_slam.server --demo --game demo-grid

Open http://127.0.0.1:8765. To use a real public ARC environment, copy .env.example to .env, set ARC_API_KEY, and run:

python -m grid_slam.server --game ft09 --operation-mode normal

The frontend is desktop-only by design.

Docker

docker build -t arc-world-model-lab .
docker run --rm -p 8765:8765 arc-world-model-lab

Visual discovery and summarization harness

vision_harness/ is a separate Python package. It imports action animations from the workbench, builds one shared ROI across all frames, and stores one human mask per target per frame. The browser annotator supports target splitting, multi-label change types, traditional forward/backward propagation, and side-by-side model overlays.

python -m pip install -e vision_harness
python -m arc_unipixel.import_workbench \
  --url http://127.0.0.1:8765 --output vision_harness/data/real --limit 12
python -m arc_unipixel.annotator \
  --data-root vision_harness/data --port 8788

Open http://127.0.0.1:8788. Real replay frames, human annotations, model weights, and provider outputs are intentionally excluded from this repository.

Experiment conclusions so far

• The complete action animation matters. Reducing an interaction to two endpoint frames loses transient motion, blinking, occlusion, and causal grouping evidence.
• A shared ROI is useful, but it is only a search window. It should not be treated as an object mask or supplied as semantic ground truth.
• Traditional tracking is strong for exact rigid motion. Integer translation and color-map matching outperform general vision models on easy pixel-exact cases, but fail on deformation, split/merge, and ambiguous appearance changes.
• UniPixel-7B can often describe the animation but is unreliable at exact ARC masks without domain adaptation. Natural-video object priors do not map cleanly to small, disconnected, discrete-color game actors.
• Tool-assisted matrix reasoning is promising. Kimi K3 and Seed 2.1 Pro can use a native grayscale matrix tool to recover precise target topology, but generated origins and masks still require deterministic matching and review.
• Semantic quality and mask quality must be measured separately. A model may correctly describe a target but emit no mask, or split one gold actor into several masks whose union is exact.
• Incomplete gold can reverse model rankings. In the 12-sample endpoint benchmark K3 scored 0.8206 mean endpoint IoU and Seed scored 0.7986, but the difference is driven by one Seed mask-generation failure and by unlabelled status-bar changes that Seed detected. Excluding that failure, Seed's endpoint score is higher. These numbers must not be interpreted as a general model ranking.

The next evaluation revision should report semantic correctness, endpoint union, full temporal coverage, target grouping, grounding reliability, and unlabelled-change coverage as separate dimensions.

See vision_harness/README.md for protocols and vision_harness/docs/annotation-guide.md for the annotation format.

Tests

make -C world_engine test build
python -m pytest -q tests/unit

cd frontend
npm ci
npm run typecheck
npm run build

cd ../vision_harness
python -m unittest discover -s tests -p "test_*.py"

Repository layout

world_engine/    C11 kernel, Python bindings, ECE DSL, MCP server
grid_slam/       map registration, masks, tracking, status bars, backend
frontend/        Preact/Vite desktop workbench
vision_harness/  annotation and multimodal visual-discovery experiments
tests/           engine and Grid-SLAM tests
docs/            architecture images

License and attribution

Original project code is licensed under Apache License 2.0. Portions derived from arcprize/arc-agi-3-benchmarking remain available under its MIT license; the original notice is preserved in LICENSES/ARC-PRIZE-BENCHMARKING-MIT.txt. See NOTICE and THIRD_PARTY_NOTICES.md.