silenthand

<p align="center"> <img src="docs/assets/silenthand-mark.svg" width="88" height="88" alt="silenthand mark"> </p>

<h1 align="center">silenthand-cua</h1>

<p align="center"> <strong>Unofficial Computer Use host for a local Mac.</strong><br> Click, type, and read AX trees without stealing focus. The MCP wrapper does not start the service. </p>

<p align="center"> <img alt="macOS" src="https://img.shields.io/badge/macOS-arm64-111?logo=apple&logoColor=white"> <img alt="Node.js" src="https://img.shields.io/badge/node-24.14.0-3c873a?logo=nodedotjs&logoColor=white"> <img alt="MCP" src="https://img.shields.io/badge/MCP-stdio-0d9488"> <img alt="CI" src="https://github.com/edosulai/silenthand-cua/actions/workflows/test.yml/badge.svg"> <img alt="License" src="https://img.shields.io/github/license/edosulai/silenthand-cua"> <img alt="npm" src="https://img.shields.io/badge/npm-not%20published-6b7280"> </p>

<p align="center"> <a href="#quick-start">Quick start</a> · <a href="#how-it-works">How it works</a> · <a href="#mcp-setup">MCP setup</a> · <a href="#available-tools">Tools</a> · <a href="#critical-constraints">Constraints</a> · <a href="#origin">Origin</a> </p>

<p align="center"> <img src="docs/assets/terminal.svg" width="760" alt="node cli.mjs doctor — BINARIES_MISSING until fetch-runtime copies ChatGPT CUA binaries"> </p>

An unofficial research host for OpenAI Computer Use. The CUA binary already speaks MCP, but over newline JSON-RPC. IDE hosts speak Content-Length. mcp-wrapper.sh is the agent-facing entry.

This is not a published npm package. Do not npx silenthand. There is no Homebrew formula. The runtime needs proprietary ChatGPT CUA binaries that must not ship — clone the repo, copy them from a local ChatGPT.app, attach mcp-wrapper.sh.

This can break whenever ChatGPT updates CUA. Bundle IDs and binary names are load-bearing. Auth bypass is a local DYLD interpose for debugging.

---

Why this exists

ChatGPT's Computer Use service can drive local apps without a Window Server focus switch. Agents still need a host that speaks Content-Length MCP, a readiness check that does not auto-start anything, and a skill that forbids Raise / open -a.

Most wrappers start a daemon and claim success from ok=true. SilentHand does neither. silenthand_doctor is read-only. Re-observe after every action.

---

Quick start

macOS 14.4+ arm64, Node 24.14.0, ChatGPT for Mac already installed:

git clone https://github.com/edosulai/silenthand-cua.git
cd silenthand-cua
nvm use          # v24.14.0
npm install

# 1. Copy SkyComputerUse* from ChatGPT.app (gitignored — never committed):
./scripts/fetch-runtime.sh
#    or: node cli.mjs fetch-runtime

# 2. Read-only doctor. Does not start the service:
node cli.mjs doctor

# 3. Copy the agent skill (default: copilot, claude, codex, agents, hermes):
node cli.mjs setup --platform hermes

# 4. Start the CUA service only if you asked for it (watchdog):
./run-silenthand.sh

Hermes attach (local wrapper — there is no npx):

printf 'Y\n' | hermes mcp add silenthand --command /ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh

(non-TTY: the CLI prompts “Enable all tools?” and cancels on EOF unless you pipe Y). Verify with both hermes mcp list and hermes config get mcp_servers. Restart Hermes / /reload-mcp so this chat actually has the tools.

REPL / one-shot (after the service is up):

node silenthand-repl.mjs
node silenthand-client.mjs list_apps
node silenthand-client.mjs get_app_state Finder
node silenthand-client.mjs click Finder --x 200 --y 300

Prerequisites

• macOS 14.4+ (arm64)
• Node.js v24.14.0 (via nvm)
• ChatGPT for Mac (source of SkyComputerUseService / SkyComputerUseClient)
• System permissions: Screen Recording + Accessibility granted for the terminal app (e.g. VS Code, Terminal.app)

./scripts/fetch-runtime.sh looks at /Applications/ChatGPT.app by default. Override with SILENTHAND_CHATGPT_APP=/path/to/ChatGPT.app.

Available Tools

Tool	Description	Required Args	Evidence notes
silenthand_doctor	Read-only readiness. Preferred first call	—	Does not start the service or touch any app
list_apps	Official sky mapper of MacComputerUseClient.listApps()	—	Observed { id, displayName, isRunning, lastUsedDate, useCount }. Not the binary MCP text.
get_app_state	A11y tree + screenshot	app	Observed; may return transient -3811 or app-specific timeouts. Pass disableDiff=true for a full AX tree.
click	Click by element_index and/or coordinates	app + (element_index or x+y)	Element-index click Observed without coords; coord-only may return noWindowsAvailable on background windows
type_text	Type text	app, text	Observed preserve-focus on background Chrome
press_key	Press keyboard key	app, key	Use super for Cmd; Command+l Observed NOT working for Chrome address bar
scroll	Scroll element	app, element_index, direction	Background element-index scroll Observed noWindowsAvailable without focus steal
drag	Drag between coords	app, from_x, from_y, to_x, to_y	Limited testing
set_value	Set element value	app, element_index, value	Browser chrome AX IDs may be invalid. WhatsApp empty composer is not settable until a letter is present.
select_text	Select text in element	app, element_index, text	Untested in matrix
perform_secondary_action	Trigger AX action	app, element_index, action	Never Raise. WhatsApp Cancel on PTT trash accepted but did not dismiss.
start_app	Host-side ComputerUseIPCAppStartRequest	app	Live result START_APP_TRANSPORT_UNAVAILABLE — signed nativePipe is not implemented. Never open -a.
silenthand_pip	Path B spectator PiP	optional live	Default dry-run. floatingPipClaimed stays false unless Observed. No windowID. Does not start ChatGPT.app.
silenthand_lock_screen_guardian	Official CUALockScreenGuardian contract	—	Extracted. Does not spawn.
silenthand_turn_ended	Official cua turn-ended CLI/IPC names	—	Extracted. Does not spawn or send a payload.

event_stream_* / computer_history_* are official extra servers when those binaries answer. Do not start a recorder unless asked.

REPL Usage

cua> list_apps
cua> get_app_state "System Settings"
cua> click "System Settings" --x 150 --y 400
cua> type_text Safari --text "hello world"
cua> press_key Safari --key Return
cua> scroll Safari --element_index 5 --direction down
cua> drag Finder --from_x 100 --from_y 200 --to_x 300 --to_y 400
cua> set_value "System Settings" --element_index 48 --value "Display"
cua> raw {"name":"click","arguments":{"app":"Finder","x":100,"y":200}}
cua> quit

How it works

MCP host (VS Code / Claude / Hermes)
  └─ Content-Length stdio → mcp-wrapper.sh → mcp-server.mjs
        ├─ silenthand_doctor / start_app / silenthand_pip /
        │  silenthand_lock_screen_guardian / silenthand_turn_ended (host-side)
        └─ NDJSON + elicitation proxy → SkyComputerUseClient
              └─ Unix socket → SkyComputerUseService
                    ↑ DYLD_INSERT_LIBRARIES=interpose_auth.dylib

The service appears to use SyntheticAppFocusEnforcer so a target can accept input without a Window Server focus switch. Inferred from binary analysis. Some Chrome routes were Observed to keep the agent’s editor frontmost. This is not a universal guarantee. Verify with NSWorkspace.shared.frontmostApplication, not Secondary Actions: Raise.

MCP setup

This is Computer Use for local apps, not a WhatsApp API. The wrapper does not start ./run-silenthand.sh. Call silenthand_doctor first. If ready=false, follow userMessage / agentNextSteps.

VS Code / Copilot (mcp.json):

{
  "servers": {
    "silenthand": {
      "type": "stdio",
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "silenthand": {
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}

Copy mcp.json.example. Do not put a real home path in the repo. After the service is up, first tool call should be silenthand_doctor, then get_app_state (pass disableDiff=true for a full AX tree), then actions. Host-side extras: start_app (Observed START_APP_TRANSPORT_UNAVAILABLE — signed nativePipe is not implemented) and silenthand_pip (Path B dry-run unless live=true), and silenthand_lock_screen_guardian (official contract, does not spawn), and silenthand_turn_ended (official CLI/IPC names, does not send a payload). list_apps is the official sky mapper, not binary MCP text. event_stream_* / computer_history_* are official extra servers when those binaries answer; do not start a recorder unless asked. Never Raise. Never open -a.

Agent skill (optional):

node cli.mjs install-skill                  # default hosts
node cli.mjs install-skill --platform hermes
node cli.mjs uninstall-skill --platform hermes

install-skill copies skills/silenthand/ into agent skill dirs. It does not start the CUA service.

Known Quirks

• element_index must be string — MCP binary treats numeric 0 as falsy. The REPL/client handle this automatically.
• click with coordinates may require the window to be accessible; pass element_index (string) for AX-based clicks that do not require foreground state. Do NOT use open -a — this violates the no-focus-steal rule and is not part of the CUA workflow.
• Multi-word app names — use quotes in REPL: get_app_state "System Settings".
• Elicitation — service sends elicitation/create. The wrapper forwards it to the MCP host. Default without a host decision is cancel, not accept. Research harnesses (silenthand-client.mjs, focus-guard.mjs) still auto-accept. SILENTHAND_AUTO_ACCEPT_ELICITATION=1 is test-only.
• Chrome background AX is chrome-only (page widgets unavailable).
• WhatsApp empty composer is not settable until it contains text. Do not type_text next to the Voice / mic button (opens a PTT draft).

Architecture

• Service: bin/macos/SilentHand CUA.app/Contents/MacOS/SkyComputerUseService (copied by fetch-runtime)
• Client: bin/macos/SilentHand CUA.app/Contents/SharedSupport/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient
• IPC Socket: ~/Library/Group Containers/2DC432GLL2.com.openai.sky.CUAService/IPC/computeruse.sock
• Interpose dylib: native/interpose_auth.dylib (built from native/interpose_auth.c)

Critical Constraints

Constraint	Reason
Bundle ID must stay com.openai.sky.CUAService	Binary checks internal host; changing causes "runtime app is missing"
Binary names SkyComputerUseService/SkyComputerUseClient must stay	Load-bearing identifiers
Service needs persistent connection OR watchdog	Idle timeout causes exit, macOS respawns ChatGPT's copy
Launch from terminal with Screen Recording	Service calls CGRequestScreenCaptureAccess() on load; TCC inheritance is observed locally but not guaranteed portable
Do not publish this package to npm	Proprietary ChatGPT CUA binaries must not ship
Do not commit bin/macos/*.app or native/*.node	./scripts/fetch-runtime.sh copies them locally

Launcher Options

./run-silenthand.sh           # watchdog mode: auto-restarts on exit
./run-silenthand.sh --once    # one-shot: exits when service dies
./run-silenthand.sh &         # background with watchdog

Permissions (TCC)

The service requests Screen Recording via CGRequestScreenCaptureAccess() in the interpose constructor. Whether it inherits the terminal's TCC grant or triggers its own prompt depends on the macOS TCC model and has not been fully verified. Ensure your terminal app has Screen Recording enabled, then check the service log for Screen Recording access: granted. Accessibility is also required for AX trees.

Origin

Extracted from ChatGPT.app Contents/Resources/cua_node/ (Codex Computer Use). See docs/UPSTREAM_PROVENANCE.md. The host MCP, packaging CLI, agent skill, and interpose source in this repo are MIT. ChatGPT CUA binaries stay on your Mac.

Status

Research host. macOS arm64 only. Not an OpenAI product. CUA binaries stay on your Mac via fetch-runtime. The MCP wrapper does not start the service. This can break on a ChatGPT.app update.

License

MIT for the host in this repository. See LICENSE and NOTICE. OpenAI’s CUA binaries, bundle IDs, and sockets are not ours and are not redistributed.