An MCP based AI Agent for PC Control

An MCP based AI Agent for PC Control is an Interactive System Control AI Agent that operates a real Windows desktop using human-like primitives only: perception (screen capture, OCR), mouse, keyboard, gestures, and navigation. No shortcut APIs — plan → act → observe → explain.

---

Why It Impresses

• Human-like interaction: Uses only MCP tools (capture, click, type, open Settings, etc.).
• True agent loop: Plan → Act → Observe → Adapt → Explain.
• Screen-grounded reasoning: OCR + UI detection to verify state.
• Explainable actions: Plain-language summaries of what changed and why.
• Safe by design: Confirmation for destructive actions; all steps logged.

---

Architecture

User intent → Planning → Perception → UI navigation → Execution → Verification → Explanation

Layer	Role
Frontend	Chat UI (demo). Primary interface: Cursor chat with MCP.
Agent	Cursor AI + system prompt + MCP tools. Plan → Execute → Observe.
Perception	Screen capture (mss/PyAutoGUI), OCR (Tesseract/EasyOCR), UI detection.
Execution	Mouse, keyboard, gestures, navigation (Start, Settings, Search, File Explorer).
Safety	log_action, ask_user_confirmation, pause/cancel.
Memory	Last screen context, last action, user preferences.

---

Requirements

• OS: Windows 10/11
• Python: 3.10+
• RAM: 8 GB min (16 GB recommended)
• Tesseract: Install Tesseract OCR and add it to PATH (or set TESSERACT_CMD).
• Cursor: Latest Cursor IDE with MCP support.

---

Setup

1. Clone and install

Option A — venv (recommended for MCP)

setup.bat

This creates .venv, activates it, and runs pip install -r requirements.txt. The project’s .cursor/mcp.json is set up to use .venv\Scripts\python.exe, so Cursor will use this environment for the MCP server.

Option B — manual

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

If you use a global Python instead of a venv, edit .cursor/mcp.json and set "command": "python" (or "py" on Windows) instead of the .venv path.

2. Tesseract (OCR)

• Download and install Tesseract for Windows.
• Ensure tesseract is on PATH, or set TESSERACT_CMD to the executable path.

3. Configure MCP Server

Project config (recommended for testing)
This repo includes .cursor/mcp.json. When you open the project folder in Cursor, the pc-control-agent MCP server is used automatically. Ensure your python (or py) has the project deps installed and is on PATH.

If python doesn’t work on Windows, edit .cursor/mcp.json and set "command": "py" instead of "command": "python".

Optional — uv

{
  "mcpServers": {
    "pc-control-agent": {
      "command": "uv",
      "args": ["run", "python", "-m", "mcp_server.server"]
    }
  }
}

Optional — global config
Use ~/.cursor/mcp.json and add the same pc-control-agent entry if you want it available outside this project. Use full paths in args if needed.

Reload Cursor (or restart) after changing MCP config.

Environment Variables

This project requires environment variables that are not included in the repository for security reasons.

1. Copy the example file:

cp .env.example .env

On Windows Command Prompt:

copy .env.example .env

Or PowerShell:

Copy-Item .env.example .env

2. Open the .env file.

3. Replace the placeholder values with your own credentials.

Example:

MISTRAL_API_KEY=hf_xxxxxxxxxxxxxxxxxxxxxxxxx

Test

1. Open the project folder in Cursor (File → Open Folder).
2. Install deps in that environment (e.g. pip install -r requirements.txt or use a venv and ensure Cursor uses it).
3. Reload Cursor (e.g. Ctrl+Shift+P → “Developer: Reload Window”) so it picks up .cursor/mcp.json.
4. Open Chat (or Composer). Under Tools, confirm pc-control-agent appears and its tools are listed.
5. Try a command, e.g.:
   • “Get screen resolution” → agent uses get_screen_resolution.
   • “Open Settings” → agent uses open_settings.
   • “Capture the screen” → agent uses capture_screen.

If the server fails to start:

• ModuleNotFoundError: No module named 'mcp' — Dependencies aren’t installed for the Python Cursor uses. Run setup.bat (or pip install -r requirements.txt in your venv), and ensure .cursor/mcp.json uses that Python (e.g. .venv\Scripts\python.exe).
• spawn python ENOENT — Cursor can’t find python. Use the full path to your Python in mcp.json (e.g. "${workspaceFolder}/.venv/Scripts/python.exe" when using the project venv).
• Check the Cursor MCP logs (Output → “MCP”) and confirm python -m mcp_server.server runs from the project root in a terminal.

---

Usage

1. Enable MCP Server in Cursor (MCP server pc-control-agent connected).
2. In Cursor chat, give a high-level goal (e.g. Show battery usage, Open File Explorer at Downloads).
3. The agent will:
   • Plan steps (e.g. Open Settings → Search “battery” → Navigate to usage).
   • Execute them one-by-one via MCP tools.
   • Observe with extract_text_from_screen / capture_screen / detect_ui_elements.
   • Explain what you’re seeing and what changed.

---

MCP Tools (overview)

• Perception: capture_screen, capture_active_window, extract_text_from_screen, detect_ui_elements, get_cursor_position, get_screen_resolution
• Mouse: move_cursor, click, double_click, right_click, click_and_hold, release_click
• Keyboard: type_text, press_key, press_key_combination, hold_key, release_key
• Gestures: swipe, swipe_from_to, two_finger_scroll, tap, long_press, pinch_in, pinch_out
• Navigation: open_start_menu, open_settings, open_search, alt_tab, switch_window, maximize_window, minimize_window, close_window, open_file_explorer, navigate_to_path
• Safety: ask_user_confirmation, log_action, pause_execution, resume_execution, cancel_execution
• Memory: store_last_screen_context, recall_last_screen_context, recall_last_action, store_user_preference, recall_user_preference, clear_context

---

Project layout

.
├── backend/           # Perception, execution, safety, memory
├── mcp_server/        # MCP server (FastMCP) exposing tools
├── frontend/          # Demo chat UI
├── agent/             # System prompt and agent docs
├── .cursor/rules/     # Cursor rules for agent behaviour
├── requirements.txt
└── README.md

---

License

MIT.