ros2_perception_mcp

ros2_perception_mcp is a dedicated, read-only-first MCP server for bounded, semantic inspection of ROS 2 perception systems.

Version 0.1.0 targets:

• Ubuntu 24.04
• Python 3.12
• ROS 2 Jazzy
• MCP Python SDK 2.x
• stdio transport

The project is intentionally designed as a dedicated perception MCP server rather than a generic ROS 2 interface.

Current Status

The current v0.1.0 development status is:

Phase 1  - Project foundation                         COMPLETE
Phase 2  - Architecture and scope                     COMPLETE
Phase 3A - Domain models                              COMPLETE
Phase 3B - Application ports and service boundary     NEXT
Phase 3  - Domain models and application ports        IN PROGRESS

Phase 3A implements and verifies the vendor-neutral perception domain model.

Focused Phase 3A verification:

12 passed

The project intentionally exposes no perception MCP tools, resources, prompts, ROS subscriptions, or physical sensor integrations yet. These capabilities are introduced only in their corresponding roadmap phases.

---

Architecture

The intended architecture is:

MCP Client
    |
    | stdio
    v
MCP Server
    |
    v
Semantic Perception MCP Surface
    |
    v
PerceptionService
    |
    +--------------------+
    |                    |
    v                    v
Domain Models      Safety / Bounds
    ^
    |
Application Ports
    ^
    |
RosPerceptionAdapter
    ^
    |
JazzyRosPerceptionAdapter
    |
    v
ROS 2 Jazzy
    |
    +----------------------+
    |                      |
    v                      v
RealSense D435i       RPLIDAR A2M8
verification          verification

Dependencies point inward.

The domain and application layers form the vendor-neutral semantic core.

ROS 2, MCP, and physical sensor integrations remain adapters around that core.

The domain layer must not depend on:

• rclpy
• ROS message packages
• tf2
• MCP SDK types
• RealSense SDKs
• SLAMTEC SDKs
• OpenCV
• device-specific APIs

The RealSense D435i and RPLIDAR A2M8 are planned physical verification devices, not public API dependencies.

---

Scope

ros2_perception_mcp owns bounded semantic inspection of ROS 2 perception systems.

The planned v0.1.0 scope includes:

• sensor discovery
• stream discovery
• semantic sensor metadata
• stream metadata
• camera metadata
• CameraInfo-derived calibration metadata
• depth metadata
• PointCloud2 metadata
• LaserScan metadata
• frame relationships
• freshness evidence
• observed rate evidence
• sensor-health evidence
• diagnostics
• explicitly bounded samples or snapshots

The MCP surface is intended to expose semantic perception operations rather than raw unrestricted ROS interfaces.

---

Explicit Boundaries

Version 0.1.0 will not expose:

• arbitrary ROS topic access
• arbitrary ROS topic publication
• arbitrary ROS service calls
• arbitrary ROS action calls
• parameter mutation
• process execution
• launch execution
• shell commands
• camera configuration
• LiDAR configuration
• LiDAR motor control
• motor control
• robot movement
• manipulator movement
• unrestricted payload forwarding
• full-rate image streaming
• full-rate point-cloud streaming

The project is read-only-first.

Inspection must not configure devices or cause actuation.

---

Responsibility Separation

The ROS 2 MCP projects intentionally have separate responsibilities.

ros2_mcp
    -> generic bounded ROS 2 inspection

ros2_control_mcp
    -> ros2_control semantics

ros2_manipulator_mcp
    -> manipulator-specific semantics

ros2_perception_mcp
    -> perception and sensor semantics

Generic ROS access belongs in ros2_mcp.

Control semantics belong in ros2_control_mcp.

Manipulator semantics belong in ros2_manipulator_mcp.

Perception-specific semantic inspection belongs in ros2_perception_mcp.

This separation prevents the individual MCP servers from becoming unbounded general-purpose robot interfaces.

---

Out of Scope for v0.1.0

The following higher-level perception and robotics capabilities are explicitly outside the v0.1.0 scope:

• object detection
• segmentation
• pose estimation
• SLAM
• Nav2
• MoveIt
• IMU support

These capabilities may be considered separately in future architecture work but are not part of the current v0.1.0 contract.

---

Phase 3A Domain Foundation

Phase 3A implements the pure Python vendor-neutral perception domain in:

src/ros2_perception_mcp/domain/

The primary implementation is:

src/ros2_perception_mcp/domain/models.py

The domain currently contains:

• SensorDescriptor
• StreamDescriptor
• CameraDescriptor
• CameraIntrinsics
• DepthDescriptor
• PointCloudDescriptor
• PointCloudField
• LaserScanDescriptor
• FrameDescriptor
• FreshnessStatus
• SensorHealth

Two finite application-owned semantic states are represented using Python 3.12 StrEnum:

• FreshnessCategory
• HealthCategory

Open-ended classifications such as sensor kinds, stream kinds, encodings, message categories, point-cloud datatypes, and frame identifiers deliberately remain extensible string values.

---

Domain Design Principles

Phase 3A follows several important design rules.

Vendor-neutral

Domain behavior does not depend on a RealSense D435i, RPLIDAR A2M8, or any other specific device.

ROS-independent

ROS messages and rclpy objects do not appear in the domain API.

ROS 2 Jazzy adapters will later convert ROS observations into semantic domain objects.

MCP-independent

Domain models do not contain MCP SDK or protocol types.

MCP is an external adapter around the application and domain layers.

Immutable

Domain models use frozen dataclasses.

Collections that belong to immutable domain values use tuples.

Incomplete metadata is representable

Unknown metadata is represented explicitly rather than fabricated.

For example, camera resolution, depth ranges, calibration information, and frame relationships may be None where appropriate.

Structural validation only

The domain validates deterministic structural invariants.

It does not invent:

• hardware limits
• vendor limits
• freshness thresholds
• rate thresholds
• physical-safety rules

---

Freshness and Health

Freshness and health are evidence-oriented.

FreshnessStatus represents:

• observation time
• age
• evidence
• optional semantic category

Freshness thresholds are not embedded in the domain model.

Threshold configuration and category derivation belong to later application and safety/bounds work.

SensorHealth represents:

• availability
• freshness evidence
• rate evidence
• findings
• semantic health category

A health result is not a physical-safety certification.

The server must never interpret sensor health as authorization for robot movement or other actuation.

---

Bounded Data

Perception systems can produce large continuous data streams.

ros2_perception_mcp is not intended to forward those streams unrestricted to an MCP client.

The intended architecture is:

Continuous ROS 2 perception stream
                |
                v
         ROS adapter observes
                |
                v
     Semantic metadata or
        bounded sample
                |
                v
          MCP response

Image, depth, point-cloud, and laser-scan access must remain explicitly bounded.

Full-rate streaming is outside the v0.1.0 scope.

---

Planned Hardware Verification

Two physical sensors are planned for later v0.1.0 verification.

RealSense D435i

Planned for:

Phase 14 - Real-hardware verification — RealSense D435i

Expected verification areas include camera, depth, calibration, stream metadata, frames, freshness, and bounded perception inspection.

RPLIDAR A2M8

Planned for:

Phase 15 - Real-hardware verification — RPLIDAR A2M8

Expected verification areas include laser-scan metadata, frames, freshness, rate evidence, health evidence, and bounded scan inspection.

These devices verify the vendor-neutral architecture.

They do not define it.

---

Run the Foundation Server

Install/synchronize the project environment:

uv sync

Run the current foundation server:

uv run ros2-perception-mcp

The process waits for MCP JSON-RPC on standard input.

At the current development stage, it intentionally advertises no perception MCP capabilities.

Set:

ROS2_PERCEPTION_MCP_CONFIG

to select an alternate TOML configuration file.

---

Development Tests

pytest is maintained as a development dependency.

The focused Phase 3A domain tests can be run with:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
uv run python -m pytest -q tests/test_domain_models.py

Verified Phase 3A result:

............                                                             [100%]
12 passed in 0.01s

Automatic third-party pytest plugin loading is disabled for this focused domain test because a ROS 2 Jazzy environment can expose unrelated ROS testing plugins such as launch_testing.

ROS-specific testing will be introduced explicitly in the corresponding later phases.

---

Project Roadmap

The v0.1.0 roadmap is:

1. Project foundation — COMPLETE
2. Architecture and scope — COMPLETE
3. Domain models and application ports — IN PROGRESS
   • Phase 3A - Domain models — COMPLETE
   • Phase 3B - Application ports and service boundary — NEXT
4. ROS 2 Jazzy adapter foundation
5. Sensor discovery and inspection
6. Camera / Image / CameraInfo
7. Depth
8. PointCloud2
9. LaserScan
10. TF / Frames / Freshness / Rate / Health
11. MCP Tools / Resources / Prompts
12. Safety bounds and diagnostics
13. Focused software verification
14. Real-hardware verification — RealSense D435i
15. Real-hardware verification — RPLIDAR A2M8
16. Final audit, documentation and v0.1.0 release readiness

Each phase requires explicit scope and must preserve the read-only, bounded, vendor-neutral architecture.

---

Documentation

Detailed development records are maintained in:

• Phase roadmap
• Phase 1 - Project Foundation
• Phase 2 - Architecture and Scope
• Phase 3 - Domain Models and Application Ports

The phase documents are intended to record not only implementation progress but also architectural decisions, explicit exclusions, validation results, and responsibility boundaries.

---

Version Assumptions

The project currently targets:

Ubuntu             24.04
Python             3.12
ROS 2              Jazzy
MCP Python SDK     2.x
MCP transport      stdio

ROS Python packages remain system dependencies and are deliberately separated from the vendor-neutral domain layer.

ROS message semantics will be verified against installed and official ROS 2 Jazzy definitions during the ROS adapter and sensor-specific implementation phases.

RealSense and SLAMTEC driver versions and conventions remain deferred until their corresponding integration and hardware-verification phases.

---

Next Step

The next development step is:

Phase 3B - Application ports and service boundary

Phase 3B will define the minimal semantic application contracts required by later ROS 2 Jazzy adapters.

It must preserve the dependency direction:

MCP Adapter
     |
     v
Application Layer
     |
     v
Domain Layer
     ^
     |
ROS 2 Adapter

Phase 3B must not introduce ROS subscriptions, hardware access, MCP perception tools, device configuration, or actuation.