websearch-mcp

Semantic web search as an MCP server, using no general search engine — no Google, Bing, DuckDuckGo, or Exa. A cheap LLM navigates the live web, a local embedding model indexes everything it walks past, and the index answers the next similar question by itself.

The idea

Building a general web index means crawling billions of pages. This does the opposite: it looks things up only when asked, and remembers what it found.

query → local vector index
  ├─ good hit  → return immediately (milliseconds, no API cost)
  └─ no hit    → navigate the live web:
        LLM picks entry sites → fetch page → LLM scores the page's links
        against the goal → expand only the most promising ones → repeat
        → index every page walked past

So it gets faster the more you use it: a brand-new topic takes a minute or two, and anything in that neighbourhood is a cache hit afterwards.

The load-bearing trick is that the model prunes at every node. A page has 50–150 outgoing links, so expanding all of them is exponential and hopeless. Scoring them against the goal turns the tree into a guided beam search — affordable precisely because the model is cheap.

Entry points

With no search engine, navigation starts from each site's own search — what a person does when they know which site holds the answer but not which page. Every entry below was probed live:

Entry point	Notes
Wikipedia (en/zh)	Article URLs directly — its /w/ search is robots-disallowed. An article on X links to the authoritative sources on X, which makes it the best general-purpose hub
GitHub repos / issues	Site search works
Hacker News	Official Algolia JSON API
Stack Exchange	Official JSON API (the HTML site rejects non-browser user agents)
MDN / PyPI / PaperMC javadoc	Site search or a fixed entry page

Sites whose search is robots-disallowed (Reddit, ArchWiki, Baidu Baike) or that are JavaScript-only shells (devdocs, the Hacker News web UI) are deliberately absent — where an official API exists, that is used instead.

Add your own with kb_add_hub, or edit websearch/hubs.py.

Tools

Tool	Purpose
web_search	Semantic search. Checks the local index first, navigates the live web on a miss
web_read	Fetch one URL's main text (and index it)
kb_status	Index size, domains covered, cache hit rate
kb_add_hub	Register a site as a future entry point

What gets stored

Per page: URL, title, summary, tags, and a 384-dimension vector — about 2KB. Page text is never retained. A hundred thousand pages fit in roughly 200MB.

Retrieval is brute-force cosine over the whole matrix (~50ms at 100k pages), so there is no ANN index to build or tune until this grows by another order of magnitude.

• index.sqlite3 — metadata
• vectors.npy + vector_ids.json — the vector matrix

Precision

Vector similarity alone is not enough. Multilingual e5 packs unrelated text into a narrow band, so an off-topic page can still score 0.80 — early versions happily answered a question about kimchi with Minecraft documentation. Retrieval is therefore followed by a cheap LLM relevance pass, and when nothing survives it says so instead of returning whatever the index happens to hold.

Being a good citizen

• robots.txt fully respected, plus a 1 req/s per-domain rate limit
• SSRF protection — hosts are resolved and private/loopback/link-local addresses refused
• Prompt injection — crawled content is always wrapped in data tags and explicitly marked as data, never instructions
• URL normalisation and content hashing to avoid re-indexing the same page

Running it

python3 -m venv .venv
.venv/bin/pip install --extra-index-url https://download.pytorch.org/whl/cpu torch
.venv/bin/pip install -r requirements.txt

cp .env.example .env      # fill in OPENROUTER_API_KEY
OPENROUTER_API_KEY=... WEBSEARCH_DATA=/var/lib/websearch .venv/bin/python server.py

Wire it into any MCP client:

{
  "websearch": {
    "command": "/opt/websearch-mcp/.venv/bin/python",
    "args": ["/opt/websearch-mcp/server.py"],
    "env": {
      "HOME": "/var/lib/websearch",
      "HF_HOME": "/var/lib/websearch/hf",
      "WEBSEARCH_DATA": "/var/lib/websearch"
    }
  }
}

Everything in websearch/config.py can be overridden by environment variable — navigation budget (NAV_MAX_PAGES, NAV_MAX_DEPTH, NAV_MAX_SECONDS), embedding model (EMBED_MODEL), politeness (PER_DOMAIN_DELAY), and so on. See .env.example.

Queries and summaries work in any language the embedding model covers; the LLM is asked to answer in the language of the query.

Cost

Embeddings run locally, so indexing is free. The LLM is used only for choosing entry sites, scoring links, and judging relevance — roughly $0.005 per cold query and nothing at all for a cache hit, using a cheap model such as DeepSeek V4 Flash through OpenRouter.

Honest limitations

• This does not replace a general search engine for arbitrary queries. It is strong in the areas you have actually used it in, and empty everywhere else.
• A brand-new topic costs one to two minutes on the first query. That is the price of navigating rather than querying a pre-built index.
• When a site redesigns or changes its robots.txt, the corresponding entry point breaks and hubs.py needs updating.

License

MIT — see LICENSE.