better-fetch

An MCP server that gives AI tools a reliable way to fetch content from the web.

Built for tools like Claude Code, ChatGPT, and any MCP-compatible client — better-fetch handles the messy reality of the modern web so your AI doesn't have to.

Why

AI tools are bad at reading the web. Pages block bots, return garbage HTML, or hide the actual content behind layers of scripts and ads. better-fetch solves this by acting as a local MCP server that fetches, renders, and extracts meaningful content from any URL.

How to Use

Claude Code

Step 1: Add better-fetch as an MCP server in your Claude Code configuration (~/.claude/claude_desktop_config.json):

{
  "mcpServers": {
    "better-fetch": {
      "command": "bun",
      "args": ["run", "/path/to/better-fetch/src/index.ts"]
    }
  }
}

Replace /path/to/better-fetch with the actual path where you cloned the repo.

Step 2: Claude Code has its own built-in WebFetch tool, so it won't automatically prefer better-fetch's web_fetch tool. To make Claude use better-fetch for web fetching, add the following to your project's CLAUDE.md (or ~/.claude/CLAUDE.md for global config):

When fetching web pages, always use the `web_fetch` MCP tool from better-fetch instead of the built-in WebFetch tool. The better-fetch tool handles anti-bot protection and JavaScript-rendered pages that the built-in tool cannot.

Once configured, you can ask Claude to fetch any URL:

Fetch the content from https://www.producthunt.com/products/notion

Debug Mode

Debug output is disabled by default. To enable detailed debug logging (network traces, cookies, page HTML, Chrome logs), set:

BETTER_FETCH_DEBUG=1

Debug data is written to the debug/ directory.

Architecture

┌─────────────────────────────────────────────────────────┐
│  AI Tool (Claude Code, etc.)                            │
│  "Fetch https://www.producthunt.com/products/notion"    │
└──────────────────────┬──────────────────────────────────┘
                       │ MCP protocol (stdio)
                       ▼
┌─────────────────────────────────────────────────────────┐
│  better-fetch MCP Server                                │
│                                                         │
│  ┌─────────────────────────────────────────────────┐    │
│  │ Tier 1: Plain HTTP fetch()                      │    │
│  │ Fast path for simple sites (example.com, etc.)  │    │
│  │ Uses browser-like headers to look legitimate.   │    │
│  │ If blocked (403/429/503) → falls through.       │    │
│  └──────────────────┬──────────────────────────────┘    │
│                     │ blocked or known anti-bot domain   │
│                     ▼                                    │
│  ┌─────────────────────────────────────────────────┐    │
│  │ Tier 2: Browser Daemon Client                   │    │
│  │ POST /fetch → Unix socket                       │    │
│  │ Spawns daemon if not running, polls /health     │    │
│  └──────────────────┬──────────────────────────────┘    │
│                     │                                    │
│                     ▼                                    │
│  ┌─────────────────────────────────────────────────┐    │
│  │ Content Extraction (defuddle)                   │    │
│  │ Strips clutter (nav, ads, sidebars, footers).   │    │
│  │ Converts cleaned HTML → Markdown.               │    │
│  │ Extracts metadata (title, description, domain). │    │
│  └──────────────────┬──────────────────────────────┘    │
│                     │                                    │
│                     ▼                                    │
│           Return clean Markdown to AI tool               │
└─────────────────────────────────────────────────────────┘
                       │
           Unix socket │ (~/.local/state/better-fetch/browser.sock)
                       ▼
┌─────────────────────────────────────────────────────────┐
│  Browser Daemon (shared, singleton process)             │
│                                                         │
│  Detached process that outlives any single MCP server.  │
│  Shared across all MCP server instances (e.g. multiple  │
│  Claude Code windows).                                  │
│                                                         │
│  ┌─────────────────────────────────────────────────┐    │
│  │ HTTP server on Unix socket                      │    │
│  │  POST /fetch   — fetch a URL in a new tab       │    │
│  │  GET  /health  — status, active/queued tabs     │    │
│  └──────────────────┬──────────────────────────────┘    │
│                     │                                    │
│  ┌─────────────────────────────────────────────────┐    │
│  │ Chrome Instance (lazy-launched on first fetch)  │    │
│  │                                                 │    │
│  │  ┌───────────┐ ┌───────────┐ ┌───────────┐     │    │
│  │  │   Tab 1   │ │   Tab 2   │ │   Tab 3   │     │    │
│  │  │  (fetch)  │ │  (fetch)  │ │  (fetch)  │     │    │
│  │  └───────────┘ └───────────┘ └───────────┘     │    │
│  │         Semaphore: max 10 concurrent tabs       │    │
│  │                                                 │    │
│  │  Challenge detection:                           │    │
│  │  If anti-bot challenge → wait for resolution    │    │
│  └─────────────────────────────────────────────────┘    │
│                                                         │
│  Lifecycle:                                             │
│  • Spawned on first browser-tier fetch                  │
│  • Auto-exits after 5 min idle (no in-flight requests)  │
│  • Graceful shutdown: drains requests, closes Chrome    │
│  • Stale socket/Chrome cleanup on next spawn            │
└─────────────────────────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────┐
│  Chrome (your existing installation)                    │
│                                                         │
│  Launched via Playwright with a FRESH user data dir     │
│  at ~/.local/state/better-fetch/chrome-profile/         │
│                                                         │
│  Your personal Chrome profiles and sessions are         │
│  NEVER touched or accessed.                             │
└─────────────────────────────────────────────────────────┘

Key Decisions

Why a shared daemon?

Multiple MCP server instances (e.g. several Claude Code windows) would each launch their own Chrome — wasting memory and fighting over the profile lock. The daemon extracts Chrome management into a single, shared process. Socket binding acts as an atomic lock: only one daemon can own the socket, so parallel spawn attempts resolve cleanly without races.

Why a real, headful Chrome browser?

Most anti-bot systems work by fingerprinting the browser environment — checking for headless indicators, automation flags, WebGL rendering, canvas hashes, and dozens of other signals. A real Chrome instance running in headful mode passes all of these checks natively because it is a real browser. No amount of header spoofing or headless patching can replicate this reliably.

Privacy: fresh profile, no access to your data

better-fetch launches Chrome using your existing Chrome installation but with a completely separate user data directory (~/.local/state/better-fetch/chrome-profile/). This means:

• It does NOT use any of your authenticated Chrome sessions
• It does NOT have access to your bookmarks, passwords, cookies, or browsing history
• It behaves like a brand-new Chrome installation with a clean slate

How persistence works

better-fetch persists two things across restarts:

1. Cookies (Chrome profile): When Chrome solves an anti-bot challenge, the resulting cookies are stored in the profile's cookie database at ~/.local/state/better-fetch/chrome-profile/. Because we use a persistent context (not incognito), these cookies survive across browser restarts — so a challenge only needs to be solved once, and subsequent visits to the same site pass through automatically.

2. Anti-bot domain list: better-fetch learns which domains require a browser. On the first fetch to any domain, it tries a plain HTTP request. If that gets blocked (403/429/503) and the browser fallback succeeds, the domain is automatically added to ~/.local/state/better-fetch/browser-domains.txt. On subsequent fetches to that domain, the server skips the plain HTTP attempt and goes straight to the browser — saving time and avoiding unnecessary blocked requests.

Concurrency

Multiple URLs can be fetched in parallel — each gets its own browser tab within the same Chrome instance. A semaphore caps concurrency at 10 simultaneous tabs to avoid overwhelming the browser or triggering rate limits. The daemon exposes active and queued tab counts via its /health endpoint.

Crash recovery and race conditions

The daemon handles real-world failure scenarios:

• Parallel spawns (thundering herd): Socket binding is atomic — one daemon wins, others detect the running instance and exit.
• Stale sockets: If the daemon crashes without cleanup, the next client detects the dead socket, removes it, and spawns a fresh daemon.
• Orphaned Chrome: On startup, the daemon checks for stale Chrome processes via SingletonLock symlinks, verifies the PID is actually Chrome (guarding against PID recycling), and kills orphans before launching.
• Shutdown overlap: The old daemon removes its socket immediately on shutdown, allowing a new daemon to bind while the old one drains in-flight requests.

Limitations

• IP-based rate limiting: Some sites use IP-based blocking in addition to browser fingerprinting, sometimes requiring CAPTCHA solving that cannot be automated. Proxy/IP rotation is not yet supported.
• Headful only: Chrome runs with a visible window. This is by design for anti-bot bypass but means it requires a display (or virtual display on Linux).

Status

Under active development.