Screen Observer MCP

A local, read-only Windows 11 screen-observation service for Claude Code and other MCP clients. It exposes a bounded, privacy-filtered, in-memory frame model through diagnostic CLI commands and eight MCP stdio tools. Observation is agent-explicit: a client decides when to start and stop collection.

Requirements

• Windows 11 for real screen capture and UI Automation.
• Python 3.12.

Development setup

py -3.12 -m venv .venv
.venv\Scripts\python -m pip install -e ".[dev]"

Generate a resolved dependency export after installing:

.venv\Scripts\python -m pip freeze --local > requirements.lock.txt

The export may contain an editable absolute Windows path for this checkout. To reproduce third-party versions in another checkout, filter that editable line and install the current checkout separately.

CLI

The installed entry point and module entry point use the same production StateService:

.venv\Scripts\screen-observer --help
.venv\Scripts\python -m screen_observer.main --help
.venv\Scripts\screen-observer status

status emits one machine-readable JSON document. start and stop only affect the collector created in that command process; this release has no daemon or cross-process IPC, so model-controlled long-running observation happens through the MCP lifecycle tools below. The read-side subcommands (snapshot, ui-tree, watch) were 1:1 duplicates of MCP tools and have been removed; use the MCP tools for the same payloads.

MCP stdio server

Start the MCP transport with:

.venv\Scripts\screen-observer mcp

It registers exactly ten tools:

• screen_observe_start — begin an agent-controlled observation session; synchronously publishes its first redacted frame, then starts background collection. The response carries a ready: true signal plus capabilities and a compact firstFrame summary so the agent can render readiness without a follow-up round trip.
• screen_observe_stop — end the session; joins the collector and clears current state, raw-window context, and every frame retained in the in-memory ring. The response carries a summary block (elapsed time, captured frames, active-window changes, last active window) so the agent can audit the observation window before exiting.
• screen_get_state
• screen_wait_for_change
• screen_wait_for_title — block until the active window title contains a substring or the deadline expires. Useful for "wait until the build terminal shows Build successful".
• screen_wait_for_idle — block until the published revision stops changing for N ms or the deadline expires. Useful for "the screen stopped updating, the task is done".
• screen_get_ui_tree
• screen_get_region
• screen_get_frame_history — pull up to N recent redacted frames from the in-memory ring
• screen_get_frame — pull one specific redacted frame by revision

The MCP server never starts screen collection merely by initializing. An agent controls the observation window by calling screen_observe_start, reading with any of the read tools for as long as needed (seconds, minutes, or until a task is complete), and then calling screen_observe_stop. Before start and after stop, every read tool returns the structured observer_not_started error. Stop is the in-memory data boundary: it clears all published frame/state artifacts immediately; it does not write them to disk.

State is JSON-only by default. screen_get_state returns image data only when include_image=true; screen_get_region is the explicit local-region image tool. The two history tools also support include_image=true and an optional bounded region. Images are privacy-filtered in source coordinates, encoded as in-memory Base64 PNG, and bounded by binary-image and complete-response limits. Frames come from a bounded in-memory ring buffer (no disk writes); see RING_DEFAULT_FRAMES, MAX_RING_FRAMES, and MAX_RING_BYTES in src/screen_observer/domain/limits.py.

Example Claude Code MCP configuration for the validated onedir artifact:

{
  "mcpServers": {
    "screen-observer": {
      "command": "C:\\project\\screen-observer-mcp\\dist\\screen-observer\\screen-observer.exe",
      "args": ["mcp"]
    }
  }
}

Replace the absolute command path if the onedir directory is copied elsewhere. A onefile artifact has not been built or validated.

Agent playbook for build/test observation

The lifecycle is fully agent-explicit. Choose one of the three workflows below — they differ only in how the agent decides that the observed task is finished. There is no "observation duration" to estimate: you start when the task starts and stop when the matching condition fires.

1. Explicit polling (screen_wait_for_change)

The simplest loop. The agent drives every step itself.

screen_observe_start         # response.ready == true, firstRevision, capabilities, firstFrame
…loop:
  screen_wait_for_change(since_revision, timeout_ms = 5000)
  inspect the result.state to decide whether the task is done
screen_observe_stop          # response.summary carries the session counters

This is the right shape when the agent knows exactly what UI element to look at.

2. Wait for a title substring (screen_wait_for_title)

Let the MCP server block until the active window title matches.

screen_observe_start
screen_wait_for_title(
    title_contains = "Build successful",
    since_revision = <firstRevision>,
    timeout_ms    = 120000)
# response.matched == true => matchedAtRevision, observedTitle
screen_observe_stop

Returns observedRedacted: true instead of erroring when the privacy policy redacted the latest window title, so a noisy environment does not crash the agent.

3. Wait for screen idle (screen_wait_for_idle)

Detect completion by absence of change — for tasks that finish without an obvious marker.

screen_observe_start
screen_wait_for_idle(idle_ms = 3000, since_revision = <firstRevision>, timeout_ms = 120000)
# response.idleReached == true => idleMs, lastObservedRevision
screen_observe_stop

idle_ms may be up to 60 000; the timeout is bounded by the shared 30 000 ms limit per call. Chain multiple calls if you need a longer overall window.

PowerShell wrapper

For humans and one-shot shell users, scripts/observe-until.ps1 encapsulates either strategy into one command and stops the MCP server on the agent's behalf:

# Block until a build/test terminal shows "Build successful":
scripts/observe-until.ps1 -WaitForTitle 'Build successful' -TimeoutSec 180

# Block until the screen stops changing for 3 s:
scripts/observe-until.ps1 -WaitForIdleMs 3000 -TimeoutSec 60

The script writes the start/stop summary to the pipeline and exits non-zero if the deadline elapses without a match.

screen_get_state and the two history tools return JSON by default; screen_get_region, screen_get_frame, and the include_image=true paths of any tool return Base64 PNG.

• The JSON paths carry everything a text-only client needs: revisions, screen geometry, active window, UI tree, and change summaries. No vision capability is required to use them.
• The PNG paths require a multimodal / vision-capable client (for example, Claude with vision) to interpret the rendered screen. Without one, the base64 payload is just opaque bytes.

If your client is text-only, prefer include_image=false (the default) and rely on the JSON contract to drive your work.

Windows onedir package

Build the reproducible PyInstaller onedir artifact from the project virtual environment:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

The expected executable is:

dist\screen-observer\screen-observer.exe

Run the packaged CLI/MCP smoke suite after the build:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\smoke_packaged.ps1

The smoke suite launches --help, status, and an MCP initialize/list/call sequence from pytest temporary working directories (gated by SCREEN_OBSERVER_PACKAGED_TEST=1). It also checks that those directories do not gain common screen image or video files. This validates the onedir artifact on the current Windows host; it is not a substitute for validation on an independent clean Windows machine.

Copy the complete dist\screen-observer directory when moving the application because the executable depends on its _internal directory. A onefile package has not been built or validated.

MCP stdout is reserved for protocol messages. Diagnostics go to stderr; tool handlers return structured safe errors without Python tracebacks.

Privacy and data lifecycle

• Screen state and the most recent published frames are retained in memory only via a bounded ring buffer; the application does not intentionally persist screenshots, videos, or screen-data history.
• Image responses are opt-in and use the same published frame/revision/privacy context as the JSON state.
• Password element names and values are removed before publication.
• Configured process, title, and physical-pixel region redactions are applied before resize and PNG encoding.
• Base64 image data and full UI text dumps are not written to diagnostic logs.
• The application cannot guarantee that Windows will never page process memory to disk.

Capture backend

DXGI Desktop Duplication (dxcam) is the production capture path; mss remains injectable for synthetic tests. The PyInstaller onedir build must collect_all("dxcam") so the frozen executable resolves the bundled DXGI/D3D11 natives on Windows 11.

Validation

.venv\Scripts\python -m pytest -q
.venv\Scripts\python -m ruff check src tests
.venv\Scripts\python -m mypy
.venv\Scripts\python -m pip check

Windows adapter, stdio protocol, and packaged-artifact coverage live in tests/adapters/test_windows_integration.py, tests/interfaces/test_mcp_server.py, and tests/integration/test_packaged_smoke.py. Run the two PowerShell scripts above to rebuild and validate the current-host onedir artifact.