CarlaMCP

Natural language → executable autonomous-vehicle scenarios in the CARLA simulator, exposed to any LLM through the Model Context Protocol.

Built on the tool-based architecture of IROSA, validated against CARLOS for containerized deployment.

A prompt like "an aggressive cut-in from the right at highway speed in heavy rain at night" becomes a validated, reproducible scenario that runs in CARLA — and can be saved as a spec for replay.

---

How it works

NL prompt ─▶ LLM picks tools ─▶ validated ScenarioManifest ─┬─▶ execute in CARLA ─▶ ScenarioResult
                                (Pydantic primitives)        │
                                                             └─▶ save .json spec ─▶ replay / reuse

Three layers, each independently testable:

Layer	Package	Responsibility
Primitives	carlamcp/primitives/	Pydantic models with LLM-facing descriptions, range constraints, and cross-field validators. The LLM never writes raw simulator code.
Executors	carlamcp/executors/	Translate validated primitives into carla API calls. The only layer that touches the live simulator.
Server	carlamcp/server.py	The MCP server exposing tools to the model.

Project structure

CarlaMCP/
├── carlamcp/
│   ├── config.py            # connection + simulation/spawn constants
│   ├── enums.py             # CARLAMap, Side, WeatherCondition
│   ├── results.py           # ScenarioResult
│   ├── carla_client.py      # CARLA binding import + get_client()
│   ├── world.py             # map / actor / sync-mode / find_ego helpers
│   ├── storage.py           # save / load scenario specs (export & replay)
│   ├── primitives/          # one file per primitive + the manifest
│   │   ├── cut_in.py  pedestrian.py  weather.py  manifest.py
│   ├── executors/           # one executor per primitive + the orchestrator
│   │   ├── cut_in.py  pedestrian.py  weather.py  manifest.py
│   └── server.py            # FastMCP server + tools + main()
├── tests/                   # pytest suite, runs WITHOUT a simulator
│   ├── fakes.py  conftest.py  test_executor.py  test_primitives.py
├── pyproject.toml  requirements.txt  .gitignore  LICENSE  README.md

---

Setup

1. Start CARLA (separate terminal)

~/carla/CarlaUE4.sh -RenderOffScreen -quality-level=Low &

2. Install CarlaMCP

cd CarlaMCP
pip install -e ".[dev]"          # package + test deps (no simulator needed)
pip install -e ".[carla,dev]"    # also installs the carla==0.9.15 binding

3. Verify the CARLA connection

python -c "from carlamcp.carla_client import get_client; print(get_client().get_server_version())"

4. Run the MCP server

python -m carlamcp.server          # or just: carlamcp

5. Wire into Claude Desktop

Add to your claude_desktop_config.json (use the absolute path to your clone):

{
  "mcpServers": {
    "carlamcp": {
      "command": "python",
      "args": ["-m", "carlamcp.server"],
      "cwd": "<path-to-your-CarlaMCP-clone>"
    }
  }
}

Restart Claude Desktop; the CarlaMCP tools will appear.

---

Usage

Once connected, talk to the model:

• "Check if CARLA is running" → get_status
• "Run an aggressive cut-in from the right at 120 km/h with 1.2s TTC" → run_scenario
• "Pedestrian crosses 25 m ahead, occluded by a parked car, in heavy fog"
• "Set the weather to rain_night at intensity 0.8" → set_weather
• "Validate this manifest before running it: {...}" → validate_scenario

Tools

Tool	Purpose
get_status	Check the simulator is reachable.
list_maps	List supported towns and their characteristics.
run_scenario	Build + validate + execute a scenario (optional save_as).
set_weather	Apply a weather preset only.
validate_scenario	Validate a manifest JSON without running it.
save_scenario	Validate and persist a manifest for later.
replay_scenario	Load a saved spec and execute it.

Scenario primitives

Primitive	Key parameters	Default map
CutInManeuver	speed_kmh, ttc_s, side, ego_speed_kmh	Town04 (motorway)
PedestrianCross	gap_acceptance_s, crossing_distance_m, occluded	Town03 (urban)
WeatherOverlay	condition, intensity	Any

Export & replay

Any scenario can be persisted as a validated JSON spec and re-run later — useful for regression sets, sharing, and scenario inpainting / augmentation pipelines:

run_scenario("...", cut_in_speed_kmh=120, cut_in_ttc_s=1.2, cut_in_side="right",
             save_as="rainy_cutin")          # writes scenarios/rainy_cutin.json
replay_scenario("rainy_cutin")               # re-executes the exact same spec

The export layer (carlamcp/storage.py) is deliberately decoupled from execution, so generated specs can be consumed by downstream tooling without a running simulator. See Roadmap → OpenSCENARIO export.

---

Testing

The suite runs entirely offline — tests/conftest.py injects a fake carla module so the executor logic (spawn geometry, velocities, ordering, error paths, collision metric) is verified without a GPU or a running simulator. This makes it CI-friendly.

pip install -e ".[dev]"
pytest -q

tests/test_executor.py doubles as the canonical example of the project's Google-style (Sphinx Napoleon) docstring convention used across every module.

---

FAQ — why Python and not C++?

CARLA runs on Unreal Engine (C++), so it's a fair question. For CarlaMCP, C++ buys nothing: the carla PyPI package is a thin binding over the C++ LibCarla client, and everything done here (spawning actors, weather, the traffic manager) is fully exposed in Python with negligible overhead at these tick rates. C++ only becomes worthwhile for work that lives inside the CarlaUE4 server build — custom Unreal sensors/actors/map assets, per-frame high-bandwidth sensor processing where the GIL is the bottleneck, or C++ ScenarioRunner extensions. None of those are on the roadmap, so CarlaMCP stays pure Python.

---

Roadmap

• [x] Phase 1 — Validated primitives + MCP server + CARLA execution
• [x] Phase 2 — Modular package, offline test suite, scenario export/replay
• [ ] Phase 3 — CARLOS Docker integration (docker-compose.yml)
• [ ] Phase 4 — CSR/ISR/TCR evaluation harness
• [ ] Phase 5 — OpenSCENARIO export for the CARLOS replay pipeline
• [ ] Paper — IEEE IV / ITSC submission

Related work

• IROSA — tool-based LLM architecture for robot skill adaptation (DLR/TUM)
• CARLOS — containerized CARLA simulation framework (ika RWTH Aachen)
• SUMO-MCP — MCP for traffic simulation (adjacent domain)

License

MIT.