rosclaw-vision-mcp

ROSClaw MCP Server for Intel RealSense RGB-D Camera via ROS2.

Part of the ROSClaw Embodied Intelligence Operating System.

Overview

This MCP server gives LLM agents eyes - the ability to see and understand their physical environment through an Intel RealSense depth camera. It implements the ROSClaw Semantic-HAL (语义硬件抽象层) pattern: the LLM never touches the raw 30Hz sensor stream, only on-demand semantic results.

RealSense Camera
      │ USB3
      ▼
realsense2_camera (ROS2, 30Hz)
      │ Fast lane → VLA/rosbag2 (data flywheel)  ← LLM never sees this
      │ Slow lane → rosclaw-vision-mcp
      ▼
LLM Agent (Claude, GPT-4o)
      │ capture_scene_snapshot() → Base64 JPEG
      │ get_object_3d_coordinates("cup") → {x:0.45, y:-0.12, z:0.05}
      ▼
rosclaw-ur-ros2-mcp
      │ pick_object(0.45, -0.12, 0.05)

Why NOT stream raw data to the LLM?

A raw RealSense stream is ~27 MB/s. MCP over JSON-RPC would crash immediately.

Instead, the LLM calls tools on-demand and receives only compressed semantic results:

• A Base64 JPEG image (~50-150 KB) when it needs to "look"
• A JSON {x, y, z} coordinate when it needs to "locate an object"
• A true/false when it needs to "check if a workspace is clear"

Features

• Snapshot capture: Base64-encoded JPEG for multimodal LLMs (Claude vision, GPT-4V)
• 3D object localization: YOLO-World detection + depth back-projection → XYZ coordinates
• Workspace collision checking: Scan a 3D volume for obstacles before arm motion
• Depth queries: Get precise distance at any pixel
• Data flywheel: Launch/stop rosbag2 recording for LeRobot VLA training data
• Thread-safe: rclpy spin in daemon thread, FastMCP in main event loop

Hardware

Field	Value
Camera	Intel RealSense D415 / D435 / D455
Interface	USB3
Protocol	ROS2 via realsense2_camera driver
ROS2 Topics	/camera/color/image_raw, /camera/aligned_depth_to_color/image_raw
Color	640×480 RGB8, 30Hz
Depth	640×480 Z16 (uint16 mm), 30Hz, aligned to color

Installation

# 1. Clone
git clone https://github.com/ros-claw/rosclaw-vision-mcp.git
cd rosclaw-vision-mcp

# 2. Source ROS2 (required)
source /opt/ros/humble/setup.bash

# 3. Install dependencies
uv venv --python /usr/bin/python3.10
source .venv/bin/activate
uv pip install -e .

# 4. Optional: YOLO-World for automatic object detection
uv pip install -e ".[detection]"

Enhanced Features (New!)

• ✅ Multi-Camera Support — Connect and control multiple RealSense cameras simultaneously
• ✅ Auto Topic Discovery — Automatically find and connect to available cameras
• ✅ YOLO Object Detection — Optional AI-powered object detection with 3D localization
• ✅ Stereo Vision — Capture synchronized images from dual cameras
• ✅ SSE Transport Mode — Persistent server for stateful connections (fixes stdio state loss)
• ✅ Systemd Service — Run as a system service with auto-restart
• ✅ Configurable Topics — Support custom ROS2 topic namespaces

Quick Start

1. Start RealSense camera(s)

Single camera:

source /opt/ros/humble/setup.bash
ros2 launch realsense2_camera rs_launch.py align_depth.enable:=true

Multiple cameras:

./scripts/launch-multi-camera.sh

2. Start MCP Server

Option A: Quick start script

./scripts/start-server.sh sse 8000

Option B: Systemd service

sudo ./scripts/install-systemd.sh $USER
sudo systemctl start rosclaw-vision@$USER

Option C: Manual

source /opt/ros/humble/setup.bash
python3 src/vision_mcp_enhanced.py --transport sse --port 8000

3. Run Demos

# Run all demos
python3 demos/demo_all.py

# Or test individual features
mcporter call rosclaw-vision.discover_cameras
mcporter call rosclaw-vision.connect_multi_camera
mcporter call rosclaw-vision.detect_objects camera_id=camera confidence=0.5

Installation

Prerequisites

• Ubuntu 22.04+
• ROS2 Humble or Jazzy
• Python 3.10+
• Intel RealSense SDK 2.0

Install Dependencies

# Clone repository
git clone https://github.com/ros-claw/rosclaw-vision-mcp.git
cd rosclaw-vision-mcp

# Install Python dependencies
pip install -e .

# Optional: Install YOLO for object detection
pip install ultralytics

# Install system service (optional)
sudo ./scripts/install-systemd.sh $USER

Claude Desktop Configuration

Stdio mode (stateless):

{
  "mcpServers": {
    "rosclaw-vision": {
      "command": "bash",
      "args": [
        "-c",
        "source /opt/ros/humble/setup.bash && python /path/to/rosclaw-vision-mcp/src/vision_mcp_server.py"
      ],
      "transportType": "stdio"
    }
  }
}

SSE mode (stateful, recommended for production):

{
  "mcpServers": {
    "rosclaw-vision": {
      "url": "http://127.0.0.1:8000/sse"
    }
  }
}

Configuration

ROS2 Topic Names

If your RealSense camera publishes to different topic names (e.g., with namespace):

# Via environment variables
export ROSCLAW_VISION_COLOR_TOPIC=/camera/camera/color/image_raw
export ROSCLAW_VISION_DEPTH_TOPIC=/camera/camera/aligned_depth_to_color/image_raw
export ROSCLAW_VISION_INFO_TOPIC=/camera/camera/color/camera_info

python src/vision_mcp_server.py

SSE Server Options

# Via command line
python src/vision_mcp_server.py --transport sse --host 0.0.0.0 --port 8080

# Via environment variables
export MCP_TRANSPORT=sse
export MCP_HOST=0.0.0.0
export MCP_PORT=8080
python src/vision_mcp_server.py

Available Tools (Enhanced Edition)

Core Tools (vision_mcp_server.py)

Tool	Description
connect_vision	Connect to camera via ROS2
disconnect_vision	Disconnect from camera
capture_scene_snapshot	Capture RGB snapshot as Base64 JPEG
get_depth_at_pixel	Get depth (meters) at a specific pixel
get_object_3d_coordinates	Detect object and get 3D XYZ position
get_scene_description	Get camera metadata and topic info
check_workspace_clear	Check if a 3D volume is obstacle-free
start_data_recording	Start rosbag2 recording for data flywheel
stop_data_recording	Stop recording and finalize bag file

Enhanced Tools (vision_mcp_enhanced.py) ⭐ New!

Tool	Description
discover_cameras	🔍 Auto-discover all available RealSense cameras
connect_multi_camera	🔗 Connect multiple cameras simultaneously
capture_from_camera	📷 Capture from specific camera by ID
get_camera_status	📊 Get status of all connected cameras
detect_objects	🎯 YOLO object detection with 3D localization
capture_stereo_image	🎬 Capture synchronized stereo pair
disconnect_all	🔌 Disconnect all cameras

Tool Usage Examples

Auto-discover cameras:

mcporter call rosclaw-vision.discover_cameras

Connect all detected cameras:

mcporter call rosclaw-vision.connect_multi_camera

Detect objects with YOLO:

mcporter call rosclaw-vision.detect_objects camera_id=camera confidence=0.5

Capture stereo image:

mcporter call rosclaw-vision.capture_stereo_image \
    left_camera=camera \
    right_camera=camera_2 \
    quality=85

Available Resources

Resource	Description
vision://status	Camera status, resolution, intrinsics
vision://topics	ROS2 topic list and types
vision://connection	Connection status

End-to-End Pick & Place Example

User: "抓起桌上的红色杯子" ("Pick up the red cup on the table")

LLM workflow:
1. capture_scene_snapshot()
   → Base64 JPEG image

2. get_object_3d_coordinates("red cup")
   → {"x": 0.45, "y": -0.12, "z": 0.38, "confidence": 0.92}

3. check_workspace_clear(x_min=0.3, x_max=0.6, ...)
   → "✓ Workspace clear"

4. [switch to rosclaw-ur-ros2-mcp]
   pick_object(0.45, -0.12, 0.38)
   → "✓ Object picked"

Object Detection

get_object_3d_coordinates() uses a two-stage strategy:

1. With YOLO-World (pip install ultralytics): Zero-shot detection - works for any object name without training. Finds bounding box, samples depth at center, back-projects to 3D.

2. Without YOLO-World: Returns the captured frame as Base64 for the LLM to visually locate the object, then the user can call get_depth_at_pixel(u, v) to get 3D coordinates.

Data Flywheel

The start_data_recording() tool captures:

• /camera/color/image_raw - RGB video
• /camera/aligned_depth_to_color/image_raw - Depth video
• /joint_states - Robot arm state
• /tf - Transforms

This data feeds the LeRobot pipeline for VLA model training (π0, OpenVLA).

Dependencies

• Python 3.10+
• ROS2 Humble or Jazzy
• mcp[fastmcp]>=1.0.0 - MCP framework
• Pillow>=10.0 - JPEG encoding (replaces heavy cv_bridge dependency)
• numpy>=1.24 - Array operations
• ultralytics>=8.0 (optional) - YOLO-World object detection

Architecture

vision_mcp_server.py
├── VisionState      - Frame data (RGB bytes, depth bytes, intrinsics)
├── StateBuffer      - Thread-safe ring buffer (10 frames)
├── VisionROS2Bridge - rclpy.Node
│   ├── _color_callback()  - /camera/color/image_raw subscriber
│   ├── _depth_callback()  - /camera/aligned_depth_to_color subscriber
│   ├── _info_callback()   - /camera/color/camera_info subscriber
│   ├── get_jpeg_base64()  - RGB → JPEG → Base64
│   ├── get_depth_meters() - Z16 depth lookup
│   ├── pixel_to_3d()      - Pinhole back-projection
│   └── check_volume_clear() - Obstacle detection
└── MCP Tools        - FastMCP @mcp.tool() definitions

License

MIT License - See LICENSE

Part of ROSClaw

• rosclaw-vision-mcp - RealSense camera (ROS2)
• rosclaw-g1-dds-mcp - Unitree G1 (DDS)
• rosclaw-ur-ros2-mcp - UR5 arm (ROS2)
• rosclaw-gimbal-mcp - GCU Gimbal (Serial)