OCR MCP

An HTTP (Streamable) MCP server written in Node.js that receives a file and a structured JSON object with empty values, extracts the file's content using local OCR / text extraction (no cloud, no API keys), and fills in the JSON values.

Supported file types:

Type	How it's read
Image	Tesseract OCR (tesseract.js) — png, jpg, gif, bmp, tiff, webp
PDF	Embedded text (pdf-parse); falls back to rendering + OCR if scanned
DOCX	Direct text extraction (mammoth)
DOC	Not supported (convert to DOCX/PDF/image)

How field filling works

Filling happens in two passes:

1. Heuristic parser (always on, local). Matches each JSON key to a label in the document text, e.g. a key invoiceNumber matches lines like Invoice Number: INV-2026 or invoice_number - INV-2026. Nested objects are supported.
2. LLM fallback (optional). For any field the heuristic could not resolve, a configured LLM is asked to fill the remaining values from the extracted text. It only fills still-empty fields and never overrides values already found. Enable it via .env (see below); leave it disabled to stay fully local.

Fields that remain empty after both passes are returned in an unfilled list.

Requirements

• Node.js 18+ (developed on v24)

Install

npm install
cp .env.example .env   # then edit .env if you want the LLM fallback

Configuration (.env)

Variable	Description
PORT	Server port (default 3000).
MCP_PATH	MCP endpoint path (default /mcp).
LLM_PROVIDER	openai | anthropic | ollama. Empty = LLM fallback disabled.
LLM_MODEL	Model name, e.g. gpt-4o-mini, claude-3-5-sonnet-latest, llama3.1.
LLM_API_KEY	Provider API key (not required for ollama).
LLM_BASE_URL	Optional base URL override.

Provider defaults for LLM_BASE_URL:

• openai → https://api.openai.com/v1
• anthropic → https://api.anthropic.com
• ollama → http://localhost:11434/v1

openai and ollama use the OpenAI-compatible Chat Completions API; anthropic uses the Messages API. Any OpenAI-compatible endpoint (Azure OpenAI, LM Studio, vLLM, etc.) works by pointing LLM_PROVIDER=openai at a custom LLM_BASE_URL.

Run

npm start
# OCR MCP server (Streamable HTTP) listening on http://localhost:3000/mcp
# LLM fallback enabled: openai / gpt-4o-mini   (or "disabled")

Health check: GET /health.

MCP tool

fill_json_from_file

Argument	Type	Required	Description
fileBase64	string	yes	File content encoded as base64.
mimeType	string	one of*	MIME type, e.g. image/png, application/pdf.
filename	string	one of*	Original filename, used to detect the type.
schema	object	yes	JSON object with the desired keys and empty values.
language	string	no	Tesseract language code(s), e.g. ind+eng or eng+fra (default ind+eng).

* Provide at least one of mimeType or filename.

Returns JSON (as text + structuredContent):

{
  "data": { "name": "Acme Corp", "invoiceNumber": "INV-2026", "missingField": "" },
  "unfilled": ["missingField"],
  "extractedTextLength": 41,
  "usedLlm": false
}

usedLlm indicates whether the LLM fallback was invoked. If a fallback call fails, an llmError field is included and the heuristic result is still returned.

Connecting from an MCP client

Point a Streamable-HTTP-capable MCP client at http://localhost:3000/mcp. Example VS Code mcp.json:

{
  "servers": {
    "ocr": {
      "type": "http",
      "url": "http://localhost:3000/mcp"
    }
  }
}

Example (raw HTTP)

# 1. initialize -> read the mcp-session-id response header
# 2. POST notifications/initialized with that session id
# 3. POST tools/call with the base64 file + schema
# See the curl handshake used in development; Accept must include
# "application/json, text/event-stream".