nvidia-vision-mcp

Give any MCP-compatible coding agent the ability to actually look at images — UI screenshots, simulator captures, design references, diagrams, screenshots of errors — through NVIDIA NIM vision models.

Works with ZCode, Claude Code, Cursor and any other MCP client, over local stdio or a self-hosted Streamable HTTP deployment. One tool, one job: analyze_image.

Why this exists

Most coding agents are text-only. When you are implementing a UI, that is a real limitation: you can describe a screenshot in prose, or paste hex values by hand, but the agent cannot actually see the thing it is building. This server closes that gap without switching to a different agent — you point it at an image, and it returns a textual analysis the agent can act on.

It is deliberately small. No dashboard, no database, no cache layer, no agent framework. Two runtime dependencies: the MCP SDK and Zod.

Architecture

MCP client  →  nvidia-vision-mcp  →  NVIDIA NIM vision model  →  analysis result
(ZCode, Claude       (stdio or          (minimaxai/minimax-m3      (text, back through
 Code, Cursor)        HTTP)              by default)                the same path)

Both transports share one code path. src/server.ts builds the MCP server and owns the vision logic; src/index.ts connects it to stdio and src/http.ts serves it over Streamable HTTP. Nothing is duplicated between the two modes.

Requirements

• Node.js ≥ 20.6
• Your own NVIDIA API key from https://build.nvidia.com/ (free tier available, starts with nvapi-)

You must supply your own key. This project is not a hosted service — there is no shared endpoint, and nothing here calls NVIDIA on your behalf. Every user runs their own instance with their own key and their own quota.

Quick start (local)

git clone https://github.com/cyril-belin/nvidia-vision-mcp.git
cd nvidia-vision-mcp
npm install
cp .env.example .env       # then put your NVIDIA_API_KEY in .env
npm run build

Verify it works before wiring it into a client:

npm test

Then add it to your MCP client. Full walkthrough: docs/local-stdio.md.

ZCode

ZCode reads ~/.zcode/cli/config.json and nests servers under mcp.servers — note that this is not the flat mcpServers key most other clients use:

{
  "mcp": {
    "servers": {
      "nvidia-vision": {
        "type": "stdio",
        "command": "/usr/local/bin/node",
        "args": ["/absolute/path/to/nvidia-vision-mcp/dist/index.js"],
        "timeoutMs": 300000
      }
    }
  }
}

Use an absolute command path (which node) — a GUI-launched ZCode does not inherit your shell PATH. Keep timeoutMs high; a detailed analysis takes 30–60 s.

Claude Code, Cursor, and other clients

{
  "mcpServers": {
    "nvidia-vision": {
      "command": "node",
      "args": ["/absolute/path/to/nvidia-vision-mcp/dist/index.js"]
    }
  }
}

Ready-to-edit copies of both are in examples/.

Self-hosted remote deployment

If you want the server reachable over HTTP instead of running locally, it also speaks Streamable HTTP — deploy your own instance to Vercel, Railway, or any container host, using your own NVIDIA key.

npm run start:http    # listens on $PORT (default 3000), endpoint /mcp

A remote instance cannot see your laptop's disk, so send the image itself: use image_url for a publicly reachable image, or image_base64 to upload the bytes directly in the call. Both work with images that exist only on the client machine. image_path still works remotely, but refers to the server's filesystem.

Railway (recommended — always-on, no request duration cap): push your fork, create a project from the repo, and Railway builds the included Dockerfile automatically. Set NVIDIA_API_KEY and MCP_AUTH_TOKEN under Variables, generate a domain, and your endpoint is https://your-app.up.railway.app/mcp. If you deploy without Docker, set the start command to node dist/http.js — the default npm start runs the stdio entry point and will exit immediately.

Vercel: import the repo and accept the detected settings; vercel.json already configures the build and the api/mcp.js function. Add NVIDIA_API_KEY and MCP_AUTH_TOKEN under Environment Variables. Two caveats — functions are capped at 60 s on Hobby (a slow analysis can be cut off), and Vercel limits request bodies to 4.5 MB, which caps image_base64 at roughly a 3.3 MB image once base64 inflation is counted. For large screenshots over image_base64, prefer Railway or another container host, or use image_url instead.

Full guide, including Docker and client configuration: docs/remote-deployment.md.

The tool

analyze_image

Argument	Type	Required	Description
prompt	string	yes	Any analysis instruction. Nothing about the prompt is hardcoded.
image_path	string	one source	Path to an image on the machine running the server. Absolute, ~/…, or relative to its cwd.
image_url	string	one source	Public http/https URL of an image to download.
image_base64	string	one source	Base64 image data, with or without a data:image/png;base64, prefix.
image_mime	string	no	Advisory MIME hint for image_base64. The real format is always sniffed.
model	string	no	NVIDIA model id override for this single call.

Exactly one of image_path, image_url or image_base64 must be supplied. Zero sources or two or more are rejected with a clear message.

Which one to use:

• Local (stdio): image_path — the server runs on your machine, so it can read your files.
• Remote (HTTP): image_url or image_base64 — the server is elsewhere and cannot see your disk. image_path still works remotely but refers to the server's filesystem.

Returns the model's textual analysis. Failures come back as MCP tool errors (isError: true) with an actionable message.

Supported formats: PNG, JPEG, GIF, WEBP. The format is detected from magic bytes rather than the file extension, so a mislabelled screenshot still works and a renamed non-image is rejected before it reaches the API.

Example prompts:

• "Analyze this UI screenshot in detail for Flutter implementation."
• "Compare this simulator screenshot with the reference design."
• "Identify layout, spacing, typography, colors, shadows and visual differences."
• "Inspect this screenshot and tell the coding agent exactly what needs to change."

Default model

The default is minimaxai/minimax-m3, the most accurate image-capable model available on integrate.api.nvidia.com at the time of writing.

It is fully configurable — set NVIDIA_VISION_MODEL, or pass model on a single call. Other verified image-capable options:

Model	Character
minimaxai/minimax-m3	Best detail and color accuracy. ~15–40 s. Default
nvidia/nemotron-nano-12b-v2-vl	Faster and lighter, less precise on exact colors.
stepfun-ai/step-3.7-flash	Reasoning model — needs a large NVIDIA_MAX_TOKENS.

Browse the catalog at https://build.nvidia.com/explore/vision. Model availability varies by account, and NVIDIA retires models periodically, so if a model starts returning 404 or 410 just point NVIDIA_VISION_MODEL at a current one — no code change needed.

Rate limits

NVIDIA's free endpoints throttle, and an agent analyzing several screenshots will fire calls in parallel — the quickest way to get a 429. This is handled for you:

• Requests are queued. At most one NVIDIA request is in flight per instance, regardless of how many calls arrive at once. Image validation still happens immediately, so a bad path fails fast instead of waiting behind a slow analysis.
• 429s are retried automatically, up to NVIDIA_MAX_ATTEMPTS (3 by default).
• Retry-After is honoured when NVIDIA sends it, in both the seconds and HTTP-date forms.
• Otherwise it backs off exponentially — 2 s, then 4 s, with full jitter.
• Retries log to stderr only, so the stdio protocol stream is never corrupted.

Two deliberate limits: a Retry-After longer than NVIDIA_RETRY_MAX_DELAY_MS (60 s) returns the error immediately rather than stalling the queue behind it, and only 429 is retried — a 403 or 404 would return the same answer on the second attempt.

Configuration

All configuration is environment variables, read from .env in the project directory or from the client's env block.

Variable	Default	Notes
NVIDIA_API_KEY	—	Required.
NVIDIA_VISION_MODEL	minimaxai/minimax-m3	Any image-capable NIM model.
NVIDIA_BASE_URL	https://integrate.api.nvidia.com/v1/chat/completions	Point at a self-hosted NIM if any.
NVIDIA_TIMEOUT_MS	180000	Per-request timeout.
NVIDIA_MAX_TOKENS	4096	Response budget.
NVIDIA_TEMPERATURE	0.2	Low = more literal descriptions.
NVIDIA_MAX_IMAGE_BYTES	20971520 (20 MB)	Applies to all three image sources.
NVIDIA_DOWNLOAD_TIMEOUT_MS	30000	Budget for fetching image_url.
NVIDIA_ALLOW_PRIVATE_URLS	false	Allow image_url to reach private/loopback addresses. Leave off unless you need it.
NVIDIA_MAX_ATTEMPTS	3	Total tries per 429. 1 disables.
NVIDIA_RETRY_BASE_MS	2000	First backoff, doubles per retry.
NVIDIA_RETRY_MAX_DELAY_MS	60000	Ceiling on any single wait.

HTTP mode only:

Variable	Default	Notes
PORT	3000	Set automatically by most platforms.
HOST	0.0.0.0	Bind address.
MCP_PATH	/mcp	Endpoint path.
MCP_AUTH_TOKEN	unset	Bearer token. Set this before exposing the server publicly.

Security

• Your API key stays yours. It is read from your environment at runtime and sent only to NVIDIA. Nothing is proxied through a third party, and the project has no telemetry.
• Never commit your key. .env is gitignored; .env.example holds placeholders only. Use your platform's secret store for deployments rather than baking keys into an image or vercel.json.
• Protect remote deployments. An unauthenticated public endpoint lets anyone spend your NVIDIA quota. Set MCP_AUTH_TOKEN; the server warns at startup if you have not.
• image_path reads the server's filesystem. Under a remote deployment, anyone who can reach the endpoint can ask it to read image files the process has access to. This is another reason to set MCP_AUTH_TOKEN.
• image_url is guarded against SSRF. Only http/https are accepted, and hosts resolving to loopback, private, link-local (including cloud metadata at 169.254.169.254), CGNAT or unique-local addresses are blocked by default. Redirects are followed manually, and every hop is re-validated so a public URL cannot bounce into your internal network. Set NVIDIA_ALLOW_PRIVATE_URLS=true only if you intentionally need internal URLs.
• Downloads and payloads are capped. NVIDIA_MAX_IMAGE_BYTES is enforced on image_url downloads (aborted mid-stream if a server omits or lies about Content-Length) and on image_base64 (checked before decoding, so an oversized payload is never allocated).
• Image contents are never logged. Only sizes and redacted URLs (origin plus path, query string stripped) appear in diagnostics.
• If a key is ever exposed, rotate it at https://build.nvidia.com/.

Testing

Everything that does not need the network — protocol, validation, rate limiting, and the HTTP transport. No API key required:

npm test

Individual suites:

npm run test:ratelimit

npm run test:http

npm run test:sources

A real end-to-end call against NVIDIA (needs a valid NVIDIA_API_KEY):

npm run smoke

Current test status

Suite	Checks	Needs a key	Covers
smoke-test.mjs --offline	12	no	stdio handshake, tools/list, schema, error paths
rate-limit-test.mjs	32	no	queueing, 429 retries, Retry-After, backoff, exhaustion
http-test.mjs	17	no	Streamable HTTP transport, shared queue, bearer auth, health
image-source-test.mjs	39	no	all three image sources, exactly-one rule, SSRF guard, size/timeout/format limits
smoke-test.mjs (full)	15	yes	all stdio checks plus a real NVIDIA vision call

100 checks pass without a key; the full smoke test passes against the live API. The tests drive the server as a real MCP client over its real transports, and use local mocks of the NVIDIA endpoint and of an image host where a live call would be slow or costly.

Troubleshooting

The server does not appear in my client. Check that you pointed it at dist/index.js (not src/index.ts) and that you ran npm run build. Use an absolute path for both the node binary and the script.

NVIDIA_API_KEY is not set. The server starts and lists tools even without a key, so you get this as a tool error rather than a crash. Confirm .env exists in the project root and contains the key, or set it in the client's env block.

403 Forbidden. NVIDIA returns 403 for both an invalid key and a valid key lacking entitlement for that model. Verify the key at https://build.nvidia.com/, then try the default model.

404 or 410 for the model. The model id is wrong, unavailable to your account, or retired. Pick a current one from https://build.nvidia.com/explore/vision and set NVIDIA_VISION_MODEL.

Requests time out in the client. Raise the client's own timeout — ZCode's timeoutMs, for example. A detailed analysis legitimately takes 30–60 s, which is longer than many default client timeouts.

Fine print in a screenshot is misread. NVIDIA downscales images server-side, so text that is small relative to a full 3x screenshot can be lost. Crop to the region of interest and send that instead — it is also faster.

Repeated 429s. You are hitting your quota rather than a burst limit. Retries will not help; wait, or check your account at https://build.nvidia.com/.

Project layout

├── src/
│   ├── server.ts       MCP server + analyze_image tool (shared by both transports)
│   ├── index.ts        local entry point — stdio
│   ├── http.ts         remote entry point — Streamable HTTP
│   ├── config.ts       environment / .env handling
│   ├── image.ts        local file resolution, validation, MIME sniffing, base64
│   ├── sources.ts      image_url download + image_base64 decode, exactly-one-source rule
│   ├── nvidia.ts       NIM request, 429 retry/backoff, HTTP error mapping
│   └── queue.ts        serial queue — one NVIDIA request at a time
├── api/mcp.js          Vercel serverless adapter (delegates to src/http.ts)
├── docs/
│   ├── local-stdio.md
│   └── remote-deployment.md
├── examples/           sample image + client configuration files
└── scripts/            test suites

License

MIT — see LICENSE.