omni-kit-mcp

Infrastructure for driving Omniverse Kit apps — Isaac Sim, Isaac Lab, or any Kit-based app — over the Model Context Protocol.

One bridge, one port, per Kit process. Projects register tools under a namespace; name collisions between projects are impossible by construction (two projects can each ship a play_scene — they land as arm.play_scene and rover.play_scene), so co-loaded projects share the single socket.

flowchart TB
    subgraph kit["Kit process (Isaac Sim · Isaac Lab · any Kit app)"]
        direction BT
        BRIDGE["<b>omni.kit.mcp</b><br/>NDJSON TCP :9009 · owner/namespace registry<br/>main-thread dispatch<br/>builtins: run_python · list_tools · reload_tools · list_bridges"]
        ARM["arm_tools/<br/>MCP_NAMESPACE = 'arm'"] -->|register| BRIDGE
        ROVER["rover_tools/<br/>MCP_NAMESPACE = 'rover'"] -->|register| BRIDGE
        PANEL["omni.kit.mcp.panel<br/>registry-driven UI"] -->|"dispatch (click == agent call)"| BRIDGE
    end
    GW["kit-mcp gateway<br/>operator view: all tools + run_python"] -->|MCP ⇄ NDJSON| BRIDGE
    GWF["kit-mcp gateway (filtered)<br/>KIT_MCP_NAMESPACE=arm · KIT_MCP_BUILTINS=0"] -->|MCP ⇄ NDJSON| BRIDGE
    CLI["kit_mcp.client<br/>scripts · tests · doctors · CLI"] -->|NDJSON| BRIDGE

Setup (once per machine)

Register the bridge permanently (afterwards, launches need no flags or env vars):

python3 scripts/install.py            # add --dry-run to preview

This edits the Kit app's persistent user.config.json (the scriptable equivalent of the Extensions window's AUTOLOAD checkbox): adds exts/ as a search path, auto-enables omni.kit.mcp + omni.kit.mcp.panel, and sets the bridge port (default 9009). Restart the app; the bridge binds with the built-in tools. --remove undoes it.

Connect an agent/IDE — install the front-end (pip install -e . or uv sync) and add to .mcp.json (Claude Code, Cursor, etc.):

{
  "mcpServers": {
    "kit": { "command": "kit-mcp" }
  }
}

No port needed — the gateway discovers a lone running bridge. Pin one with "env": { "OMNI_KIT_MCP_PORT": "9009" } when several Kit processes run.

That alone is fully usable: run_python executes arbitrary Python on Kit's main thread (persistent sessions supported), which reaches everything in the sim. Tools discovered from the bridge appear as native MCP tools (robot_demo.spawn_robot → robot_demo__spawn_robot; dots are illegal in MCP names).

Adding a project (the contract)

A project is a plain Python package — not a Kit extension. It declares a namespace and exposes one entrypoint (copy examples/demo_tools/):

# ~/myproj/isaac/myproj_tools/__init__.py
MCP_NAMESPACE = "robot_demo"                       # defaults to the module name

def register(registrar):                   # receives an owner-bound registrar
    @registrar.tool("Spawn a robot into the scene.", {"name": {"type": "string"}})
    def spawn_robot(name: str = "robot_1"):
        import omni.usd                    # Kit imports live inside handlers
        ...
        return {"path": f"/World/{name}"}  # payload only — no envelope

Register it persistently (once per project):

python3 scripts/install.py add-project --path ~/myproj/isaac --module myproj_tools

Next launch, the bridge advertises robot_demo.spawn_robot; the panel grows a robot_demo section; agents see it after refresh_tools. Per-launch override instead of persistence: KIT_MCP_TOOL_PATHS=~/myproj/isaac KIT_MCP_TOOL_MODULES=myproj_tools.

One registration, three surfaces. A registered tool isn't just an agent verb: the control panel renders it as a clickable widget automatically, and the reference client/CLI can call it from scripts — all three go through the same dispatch entrance. Write the tool once; button, agent call, and script call come with it.

Handler contract: params arrive as kwargs; return a JSON-serializable payload (None → {}) or raise (ToolError attaches diagnostics like captured stdout). Sync or async. The bridge owns the envelope and the main-thread hop.

Curated per-project agent views — same binary, same socket, different env:

"robot-demo-agent": {
  "command": "kit-mcp",
  "env": { "OMNI_KIT_MCP_PORT": "9009",
           "KIT_MCP_NAMESPACE": "robot_demo",
           "KIT_MCP_BUILTINS": "0" }
}

exposes only robot_demo.* — no run_python — for agents that should stay on the stable API. run_python is the operator's escape hatch and how tools are prototyped; registered tools are what agents rely on.

Scripting the bridge (no MCP needed)

The bridge's wire protocol is not MCP — the gateway above is the only MCP speaker. For scripts, tests, and health checks, use the reference client (stdlib-only, single-file, also copyable as-is):

from kit_mcp.client import call, BridgeClient, BridgeError
call("demo.ping", {"message": "hi"})                 # one-shot; port auto-discovered
with BridgeClient() as c:                            # persistent + reconnect
    c.call("run_python", {"code": "result = 1"})     # (pass port=... to pin)

python3 -m kit_mcp.client <tool> ['{"json":"params"}'] [--port N]
python3 -m kit_mcp.client --list-bridges          # what's running
# exit codes: 0 success · 1 bridge answered with an error · 2 unreachable

Port resolution — you usually don't specify one. Each running bridge advertises itself in a per-user runtime dir ($XDG_RUNTIME_DIR/omni-kit-mcp/). A client resolves its port as: explicit port=/--port → OMNI_KIT_MCP_PORT → auto-discovered when exactly one bridge is running → a loud error listing candidates when several are (e.g. the app on 9009 and a Lab run on 9010 — then name one). The single-process common case is zero-config; only genuinely ambiguous multi-process boxes ever need a port. (Agents via the gateway are even simpler — the gateway resolves once at startup and they just call tools.)

import kit_mcp.client works without the mcp SDK installed — only the gateway needs it.

Hot reload

Edit tool code, then (via any client): reload_tools {"module": "myproj_tools"} → the owner drains in-flight work, the module tree re-imports (stale bytecode purged), tools re-register. Live handles that must survive (articulation views, physics handles) belong in a submodule named *_state — state modules are preserved across reloads. Then refresh_tools on the MCP front-end picks up schema changes. No Kit restart.

Control panel

omni.kit.mcp.panel renders the live registry: one section per namespace, parameter widgets generated from each tool's schema. Every button goes through bridge.dispatch(...) — the same entrance a socket request takes — so a human click and an agent call are the same code path by construction. Projects that want bespoke UI ship a thin widgets-only extension obeying the same rule.

Wire protocol (NDJSON over TCP)

One JSON object per \n-terminated line, each direction:

→ {"type": "robot_demo.spawn_robot", "params": {"name": "robot_1"}}
← {"status": "success", "result": {"path": "/World/robot_1"}}
← {"status": "error", "message": "...", "traceback": "...", ...}
→ {"type": "list_tools", "params": {}}
← {"status": "success", "result": {"tools": {"robot_demo.spawn_robot": {"description": ..., "parameters": ..., "namespace": "robot_demo", "owner_id": ...}}}}

A malformed line gets one error response; the connection stays healthy. Many clients may hold connections concurrently; each is isolated. Tool execution is serialized on Kit's main thread (USD/PhysX are not thread-safe).

Configuration reference

Knob	Persistent (carb setting under /persistent/exts/omni.kit.mcp/)	Per-launch env override
Bridge port	autostartPort	OMNI_KIT_MCP_PORT
Bind address	autostartHost	OMNI_KIT_MCP_BIND
Tool packages	toolModules	KIT_MCP_TOOL_MODULES
Package paths	toolPaths	KIT_MCP_TOOL_PATHS

Remote access. The bridge binds localhost by default — it serves run_python (remote code execution by design), so listening beyond loopback is an explicit per-box decision. Options: an SSH tunnel (zero config), or set the bind address to a private/tailnet IP (install.py --bind-host <ip>; never 0.0.0.0 on a public host). Remote callers can't read the box's portfiles, so the list_bridges builtin exists: ask any reachable bridge (typically the stable installed-app port) and it reports every bridge on its box — including ephemeral-port siblings.

Front-end (kit-mcp) env: OMNI_KIT_MCP_PORT (optional — auto-discovered when exactly one bridge runs; set it to pin a process), OMNI_KIT_MCP_HOST, KIT_MCP_NAMESPACE (comma allowlist), KIT_MCP_BUILTINS (0 hides run_python/reload_tools), KIT_MCP_NAME / KIT_MCP_INSTRUCTIONS.

Isaac Lab (standalone scripts)

The bridge's only Kit dependency (omni.kit.async_engine) ships in Kit's core extension set, so it runs in any Kit app — including Isaac Lab. Lab standalone scripts (train/play) differ from the Isaac Sim app in three ways: the script owns the Kit app (AppLauncher + Lab's own isaaclab.python.*.kit experiences), so persistent registration doesn't apply — enable the bridge per-script via --kit_args; the script owns the main loop, so bridge commands execute only while the app is pumped (env.step() / sim.step()) — a script blocked between steps stalls dispatch until the next pump; and configuration is env-var only. Note the port: "one bridge, one port, per Kit process" — a Lab standalone run is its own Kit process, so when it runs alongside the installed app (which holds 9009) it needs its own port (9010 by convention). See examples/isaaclab_bridge_launch.py for the template.

Layout

exts/omni.kit.mcp/          the bridge extension (protocol, bridge, builtins, autoload)
exts/omni.kit.mcp.panel/    registry-driven control panel (optional)
kit_mcp/                    the MCP front-end (FastMCP server, TCP client to the bridge)
scripts/install.py          permanent registration: base install + add-project
examples/demo_tools/        the template tool package — copy to onboard a project
tests/                      protocol / registry / autoload / live-socket tests (no Kit needed)

Development

python3 -m pytest tests/    # the socket server runs under plain asyncio in tests