mcp-nv-segment-ctmr

A FastMCP server exposing NV-Segment-CTMR (NVIDIA's VISTA3D-based 3D medical image segmentation foundation model) as MCP tools, deployed via Prefect Horizon. GPU inference is dispatched to a Modal serverless L40S endpoint, so the container needs no GPU, no MONAI, and no model weights.

Research use only. NV-Segment-CTMR is released under the NVIDIA OneWay Non-Commercial License. It is not a cleared diagnostic device and its outputs must not drive patient care decisions.

Why this one is different

The other servers in this family (locate-anything, fundus-agent, MIRAGE) pass images as base64 in a tool argument. That does not work here:

	2D fundus / CXR	3D CT / MR
Typical payload	0.3–3 MB	50–200 MB (.nii.gz, already compressed)
Runtime	2–20 s	1–10 min (sliding window over 117 class prompts)
Output	a few boxes	a mask the same size as the input

So the contract inverts. Volumes are passed by URI, the mask is written server-side and returned as a download URL, and long jobs use submit/poll rather than a blocking request. What crosses the MCP boundary is structure names in and volumes in mL out — things an agent can actually reason over.

Architecture

MCP client ──► Horizon (this container, CPU)
                 │  name→index resolution, URI preflight, job brokering
                 ▼
              Modal ASGI app (CPU)  ──►  SegmentCTMR class (L40S)
                 │  /jobs /labels /files      VISTA3D sliding-window inference
                 ▼
              Modal Volume  ──►  mask .nii.gz download URL

The Modal side lives in modal_nv_segment_ctmr.py; see download_nv_segment_ctmr.py for fetching the HF repo and the volume setup steps.

Tools

Tool	Description	Returns
list_anatomical_structures	Search the 345+ class vocabulary	Name → index map
check_image_uri	HEAD preflight on a NIfTI URI	Reachability, size
segment_structures	Segment named organs; submits and polls	Volumes + mask URL
segment_everything	Whole-body/brain pass; always async	call_id
get_segmentation_status	Poll a running job	Status or full result
health	Liveness probe, includes upstream check	Status

Typical agent flow

list_anatomical_structures(query="kidney")
  → {"kidney_right": 14, "kidney_left": 5, "kidney_cyst_left": 116, …}

check_image_uri("https://…/abdomen_ct.nii.gz")
  → {"success": true, "size_mb": 62.4, "filename_looks_like_nifti": true}

segment_structures(
    image_uri="https://…/abdomen_ct.nii.gz",
    study_id="case_0042",
    structures=["liver", "spleen", "kidney_left", "kidney_right"])
  → {"structures": {"liver": {"volume_ml": 1487.2, …}, …},
     "mask_download_url": "https://…/files/case_0042-17.../case_0042_seg.nii.gz"}

segment_everything returns a call_id immediately; feed it to get_segmentation_status (optionally with wait_s) until it reports done.

Design notes

Name resolution happens here, not on the GPU. The class vocabulary is fetched once from Modal's CPU tier and memoized for an hour. Matching is case-insensitive exact, then normalized, then unique-substring — so "left kidney", "kidney_left", and "Kidney Left" all resolve. Ambiguous or unknown names come back as a validation error with a pointer to list_anatomical_structures rather than a MONAI stack trace.

Prompt validation is local. Indices are range-checked and filtered against the checkpoint's unsupported set (16, 129–131, 133, 137–145, 162) before anything is dispatched, so a bad request never costs GPU time.

Results are trimmed. A 117-class result is a lot of tokens to spend on organs with 0.2 mL of predicted volume. Structures are ranked by volume and capped at max_structures (default 40); bounding boxes are opt-in.

MRI_BRAIN has a guard. The checkpoint only supports skull-stripped, intensity-normalized T1. Running it on a raw T1 produces confident nonsense, so brain_preprocessed=True is required as an explicit acknowledgment.

Configuration

Variable	Required	Description
MODAL_API_URL	yes	Modal ASGI base URL, e.g. https://mathgcloud--nv-segment-ctmr-api.modal.run
DEFAULT_WAIT_S	no	Seconds segment_structures polls before returning a call_id (default 240)
FASTMCP_DOCKET_URL	no	Redis URL (rediss://…) for background tasks

Connecting

{
  "mcpServers": {
    "nv-segment-ctmr": {
      "url": "https://nv-segment-ctmr.fastmcp.app/mcp"
    }
  }
}

Running

# Docker
docker build -t nv-segment-ctmr-mcp .
docker run -p 8080:8080 \
  -e MODAL_API_URL=https://<your-modal-app>.modal.run \
  nv-segment-ctmr-mcp

# Local
pip install -r requirements.txt
export MODAL_API_URL=https://<your-modal-app>.modal.run
python server.py

Multi-stage python:3.11-slim build, runs as a non-root user, exposes port 8080. Serves over stateless HTTP with JSON responses.

License

Apache License 2.0 for this code — see LICENSE. The underlying model is NVIDIA OneWay Non-Commercial.