webharvest

A local Firecrawl alternative: scrape and search over MCP. Query search engines, fetch and parse web pages, all via Claude's MCP protocol.

What It Does

webharvest is a daemon that:

• Runs a local Model Context Protocol server
• Exposes search (via SearXNG) and web scraping (via Playwright)
• Integrates with Claude Code as an MCP server
• Enforces politeness: per-domain request queueing (2 concurrent, 500 ms apart)
• Caches results in SQLite for 24 hours by default

You control the browser automation — no opaque API calls, no subscription, all on your machine.

Installation

Prerequisites

• Node 20+
• Docker & Docker Compose (for SearXNG search)
• Chromium (installed via Playwright, see below)

Steps

1. Clone and build:

   git clone <repo-url>
   cd webharvest
   npm install
   npx playwright install chromium
   npm run build
   

2. Install the CLI globally:

   npm link
   

   This makes webharvest available as a command in your shell. Alternatively, use npx webharvest to run without installing globally.

3. Start SearXNG (search backend):

   # Create your local config from the template (one-time).
   # searxng/settings.yml is gitignored: secret_key is per-install and must not be committed.
   cp searxng/settings.yml.example searxng/settings.yml
   sed -i '' "s/CHANGE_ME_TO_A_RANDOM_STRING/$(openssl rand -hex 32)/" searxng/settings.yml
   
   # Start the container
   docker compose up -d
   
   # Verify it's up (wait 10 seconds, then):
   curl -s 'http://127.0.0.1:8080/search?q=hello&format=json' | head -c 100
   

   The last command must return JSON. If it returns HTML or a 403, formats in searxng/settings.yml is missing json — fix it and restart the container. Note that limiter must stay false, or SearXNG rate-limits the daemon querying it.

4. Register the daemon with launchd (macOS only for now):

   webharvest install
   

   This registers the daemon with launchd, so it starts automatically and restarts if it crashes. Use webharvest start to start it immediately, or it will start at next login.

5. Connect to Claude Code:

   claude mcp add webharvest -- node /absolute/path/to/webharvest/dist/mcp/index.js
   

   Replace the path with the actual path to your clone. You can find it with pwd in the webharvest directory.

Automated Setup (hand this prompt to an AI agent)

If you'd rather not run the steps above by hand, paste the prompt below into an agent that can execute shell commands and edit files (Claude Code, Codex, Cursor, Gemini CLI, …). It performs the same installation and then registers webharvest as an MCP server in whichever agents you have installed.

Read it before you run it: it installs a launchd job, starts a Docker container, and edits your agents' MCP config files.

You are setting up "webharvest" — a local MCP server that gives you web search
and web scraping. Work through the steps in order, run every verification
command, and stop and report if one fails instead of continuing.

Repository: the directory you were pointed at, or clone it if I gave you a URL.
Let REPO be its absolute path (get it with `pwd`, never guess, never use `~`).

1. Prerequisites. Check `node --version` (must be 20+), `docker --version`, and
   `docker compose version`. If Docker is not installed or its daemon is not
   running, tell me and stop — search will not work without it.

2. Build.
     cd REPO && npm install && npx playwright install chromium && npm run build
   Verify REPO/dist/mcp/index.js, REPO/dist/cli/index.js and
   REPO/dist/daemon/index.js now exist.

3. Search backend (SearXNG). If REPO/searxng/settings.yml does not exist:
     cp searxng/settings.yml.example searxng/settings.yml
   then replace the literal CHANGE_ME_TO_A_RANDOM_STRING in it with the output
   of `openssl rand -hex 32`. Never commit this file — it is gitignored on
   purpose. In that file confirm `formats` contains `json` and `limiter` is
   `false`; fix them if not. Then:
     docker compose up -d
   Wait ~10 s and verify search actually answers with JSON:
     curl -s 'http://127.0.0.1:8080/search?q=hello&format=json' | head -c 100
   HTML or a 403 back means the settings above are wrong. Fix and restart the
   container; do not move on.

4. Daemon. Run `npm link` in REPO so the `webharvest` command exists (if that
   needs sudo on this machine, tell me instead of running sudo yourself).
   On macOS: `webharvest install` — this already loads the launchd job and
   starts the daemon, so run `webharvest status` next and only call
   `webharvest start` if it reports the daemon is not running.
   On Linux/Windows there is no launchd integration — start the daemon in the
   background yourself with `node REPO/dist/daemon/index.js` and tell me it
   will not survive a reboot.
   Verify: `curl -s http://127.0.0.1:8787/health` returns JSON.

5. Register as an MCP server. The MCP entry point is always:
     command: node
     args:    ["REPO/dist/mcp/index.js"]
   It talks to the daemon over http://127.0.0.1:8787 and does not start it, so
   step 4 must be working first. If the daemon runs on another port, also set
   the env var WEBHARVEST_URL for the server.

   Do this for each agent that is actually installed on this machine — check
   with `command -v claude`, `command -v codex`, … and not by the existence of
   a ~/.something directory, which is often left behind by an uninstall. Skip
   the rest, and never overwrite an existing "webharvest" entry without
   telling me:

   - Claude Code:
       claude mcp add -s user webharvest -- node REPO/dist/mcp/index.js
     `-s user` matters: without it the server is registered only for the
     directory you happen to be in.
   - Codex: add to ~/.codex/config.toml
       [mcp_servers.webharvest]
       command = "node"
       args = ["REPO/dist/mcp/index.js"]
   - Cursor (~/.cursor/mcp.json), Windsurf, Claude Desktop, and most others
     use the same JSON shape — merge into the existing "mcpServers" object,
     do not replace the file:
       { "mcpServers": { "webharvest": {
           "command": "node", "args": ["REPO/dist/mcp/index.js"] } } }
   - Gemini CLI: same JSON shape in ~/.gemini/settings.json.

   If an agent you find is not in this list, look up its MCP config location in
   its own docs and use the same command/args.

6. Verify end to end. Talk to the MCP server directly over stdio, so you do
   not have to restart anything to know it works:
     printf '%s\n' \
       '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}' \
       '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
       '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search","arguments":{"query":"model context protocol","limit":3}}}' \
       | node REPO/dist/mcp/index.js 2>/dev/null | tail -1
   That must come back with real search results, not "Не удалось" — the latter
   means the daemon or SearXNG is not reachable, so go back to step 3 or 4.
   Then repeat with a `scrape` call on one of the returned URLs.
   Also confirm the registration itself: `claude mcp list` for Claude Code
   (expect "webharvest … ✔ Connected"), `codex mcp list` for Codex. For the
   others, tell me to restart the agent and check that the tools `scrape` and
   `search` appear.

7. Report: what you installed, the exact paths and config files you touched,
   what still needs a restart, and anything you skipped and why.

Common Commands

• webharvest start — Start the daemon manually
• webharvest stop — Stop the daemon
• webharvest status — Check if it's running
• webharvest logs — Follow daemon logs (Ctrl+C to exit)
• webharvest uninstall — Remove launchd registration

What It Can't Do

webharvest has clear limitations. Before opening an issue, check this list:

❌ No PDF Support

PDFs are not parsed. The scraper detects the non-HTML content type and returns a clean not_html error — it never hands you raw binary data. Use a dedicated PDF parser if you need full-text extraction from PDFs.

❌ No Authentication

Pages that require login are not supported. The scraper has no credential storage, session management, or form-filling logic for authentication. If a page redirects to a login screen, the scraper will return the login page HTML, not the authenticated content.

❌ Interactive Anti-Bot Challenges

Challenges like Cloudflare Turnstile, hCaptcha, or DataDome are not solved. These are reported with status blocked and an explanation in the response. There is no integration with captcha-solving services.

❌ robots.txt Is Not Enforced as a Prohibition

The scraper does not read or honour robots.txt as a legal prohibition. Instead, politeness is enforced through rate limiting:

• Per-domain request queue: at most 2 concurrent requests per domain, 500 ms apart
• This is respectful by default: delays stack, so successive requests to the same domain are naturally throttled

The brief specifies why robots.txt as a prohibition was deliberately omitted: it is a machine-readable convention, not a legal boundary, and enforcing it in code would create false confidence. Rate limiting is more honest: you are making fewer, slower requests, which is measurably less disruptive than hitting a site at full speed.

A respectRobots config flag was considered and removed. It is not coming back. Rate limiting is the mechanism.

❌ SSRF Protection Doesn't Close DNS Rebinding

The private-address check (assertPublicHost) resolves the hostname and validates the addresses it gets, but that's a check-then-use, not a closed gate: undici and Playwright each resolve the hostname again, independently, when they actually connect. A host with a short DNS TTL that answers with a public address during the check and a private one moments later (classic DNS rebinding) can walk past the guard. Properly closing this needs pinned-IP dispatch — resolve once, force every subsequent connection onto that exact address, and validate the connecting socket's remote address before reading any bytes — which is real engineering, not a quick patch, and is out of scope for a personal tool. Known and accepted, not silently assumed away.

Configuration

Settings live in ~/.webharvest/config.json. The daemon only ever reads this file — it is never created for you, so if you want to override defaults, create it yourself:

{
  "port": 8787,
  "cachePath": "/Users/you/.webharvest/cache.db",
  "cacheTtlMs": 86400000,
  "searxngUrl": "http://127.0.0.1:8080",
  "braveApiKey": null,
  "idleTimeoutMs": 300000
}

Note: cachePath is used as-is, with no ~ expansion — write an absolute path (as above), not ~/.webharvest/cache.db, or you'll get a literal ./~ directory relative to wherever the daemon happens to be started from.

Environment variables override config file settings:

• WEBHARVEST_PORT — daemon listening port
• WEBHARVEST_SEARXNG_URL — search backend URL
• BRAVE_API_KEY — optional Brave Search API key (for fallback search)

Note: host and allowPrivate cannot be set from config.json for security reasons. The daemon always binds to 127.0.0.1 only.

Architecture

• src/core/ — Scraping logic: browser pool, SSRF protection, cache
• src/daemon/ — HTTP server (localhost only) with health check
• src/mcp/ — Model Context Protocol server (stdio-based)
• src/cli/ — Command-line interface for daemon management
• searxng/ — Docker Compose config and SearXNG settings

Development

npm test          # Run unit tests
npm run typecheck # TypeScript strict mode check
npm run build     # Compile to dist/

Tests are excluded from the live-test suite unless you run npm run test:live.

License

[Check LICENSE file]