screen-mcp

Screen-aware MCP (Model Context Protocol) server. Captures the user's screen, finds text-worthy regions via heuristic analysis, runs OCR, and returns a compact JSON (~500 tokens) instead of a full screenshot (~500,000 tokens).

Architecture

Module	Description
screenshot.py	Fast screen capture via mss
detection.py	Heuristic text-region finder (edge density + contrast scoring)
ocr.py	Tesseract OCR wrapper
main.py	MCP stdio server + HTTP debug server + ScreenWatcher background monitor

Three Modes

1. MCP stdio (for Codex / Claude Desktop / Cursor)

Add to your MCP config (mcp.json):

json { "mcpServers": { "screen": { "command": "python", "args": ["-m", "src.main"], "cwd": "/path/to/screen-mcp" } } }

Three tools are exposed:

• screen_analyze - Full analysis: capture -> detect regions -> OCR
• screen_watch - Background monitoring: start / stop / status / force_check
• screen_diff - One-shot change detection

2. Demo Mode

ash python -m src.main --demo

Outputs a JSON analysis of the current screen to stdout.

3. HTTP Debug Server

ash python -m src.main --http-port 8000

• GET /health returns {"status": "ok"}
• POST /analyze returns full screen analysis JSON

Installation

`ash pip install mss Pillow pytesseract numpy fastapi uvicorn

Install Tesseract OCR separately

Windows: winget install UB-Mannheim.TesseractOCR

macOS: brew install tesseract

Linux: sudo apt install tesseract-ocr

`

Output Example

json { "monitors": [{"left": 0, "top": 0, "width": 1920, "height": 1080}], "regions": [{"x1": 420, "y1": 140, "x2": 1160, "y2": 460, "confidence": 0.76}], "texts": [ {"text": "New chat", "confidence": 0.96, "bbox": [43, 54, 100, 64]}, {"text": "Search", "confidence": 0.96, "bbox": [43, 84, 82, 94]} ], "summary": "Found 5 text regions | Text (118): New chat, Search, Plugins..." }

Token Savings

Method	Tokens
Raw screenshot (1920x1080 PNG)	~500,000+
screen-mcp JSON	~500-2,000
Savings	99.6%

Configuration

Parameter	Default	Description
--min-area	3000	Min pixel area for text region detection
--ocr-conf	0.4	Minimum OCR confidence threshold
--monitor	1	Monitor index (1=primary)

Notes

• OCR uses Tesseract with English language pack by default
• Chinese OCR requires installing chi_sim.traineddata
• The ScreenWatcher background monitor uses MSE-based frame differencing with configurable interval and threshold
• Tesseract auto-detects common installation paths; falls back to PATH lookup