<p align="center"> <img src="docs/assets/banner.png" alt="webdocs-mcp" width="100%" /> </p>

<p align="center"> <a href="https://github.com/saianthireddy/webdocs-mcp/actions/workflows/ci.yml"><img src="https://github.com/saianthireddy/webdocs-mcp/actions/workflows/ci.yml/badge.svg" alt="CI"></a> <img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="Python"> <img src="https://img.shields.io/badge/license-MIT-green" alt="License"> </p>

🩺 webdocs-mcp

Crawl any documentation site, index it into DuckDB with hybrid (semantic + BM25) search, and expose the whole thing to LLM agents as an MCP server — so your coding assistant reasons over current docs instead of its training cutoff.

Runs completely offline by default: the built-in hashing embedder needs no API key, and every test runs without touching the network. Set one environment variable to switch to OpenAI embeddings in production.

Why

LLM agents hallucinate stale APIs. The fix is retrieval over the actual docs of the actual version you use. webdocs-mcp is the smallest honest stack that does this end to end: point it at a docs site, it crawls with hierarchy tracking, chunks and embeds every page, and serves search_docs / get_doc_page tools over the Model Context Protocol to Cursor, VS Code, or Claude Code.

Architecture

flowchart LR
    U["You / CI"] -->|POST /fetch_url| API["FastAPI"]
    API --> JOBS["Job runner<br/>(background threads)"]
    JOBS --> CRAWL["Crawler<br/>BFS, same-domain,<br/>parent/child hierarchy"]
    CRAWL --> CHUNK["Chunker<br/>paragraph-aware + overlap"]
    CHUNK --> EMB["Embedder<br/>hashing (offline) / OpenAI"]
    EMB --> DB[("DuckDB<br/>pages + chunks + embeddings")]
    AGENT["LLM agent<br/>(Cursor / VS Code / Claude Code)"] -->|MCP JSON-RPC| MCP["/mcp endpoint"]
    MCP --> SEARCH["Hybrid search<br/>cosine + BM25"]
    SEARCH --> DB
    BROWSER["Browser"] -->|/map| MAPS["Site maps<br/>pure HTML tree"]
    MAPS --> DB

Design choices worth calling out:

• DuckDB over a vector-DB server — a doc index for one team is thousands of chunks, not billions. One portable file, zero ops.
• BM25 + embeddings, always both — embeddings catch paraphrases ("how do I get my money back" → refund policy); BM25 catches exact identifiers (ERR_LOCK_TIMEOUT) that embeddings smear. Scores are min-max normalised and blended 60/40.
• Injectable fetcher — the crawler takes a plain (url) -> html callable. Tests inject a dict-backed fake site; production uses httpx; a headless browser slots in without touching crawl logic.
• MCP implemented directly — the streamable-HTTP core of MCP is JSON-RPC dispatch (initialize, tools/list, tools/call). Owning those ~100 lines keeps the stack dependency-light and fully offline-testable.
• Threads now, Redis when needed — jobs run on daemon threads behind a JobManager interface shaped like a queue consumer. docker-compose already ships the Redis service for the scale-out path.

Quickstart

git clone https://github.com/saianthireddy/webdocs-mcp.git
cd webdocs-mcp

python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

pytest            # 23 tests, fully offline
ruff check src tests

PYTHONPATH=src uvicorn webdocs.api:app --port 9111

Then:

# start a crawl
curl -X POST localhost:9111/fetch_url -H 'Content-Type: application/json' \
     -d '{"url": "https://docs.astral.sh/ruff", "max_pages": 30}'

# watch progress
curl localhost:9111/job_progress

# search what was indexed
curl 'localhost:9111/search_docs?query=how+do+I+ignore+a+rule&top_k=3'

Interactive OpenAPI docs at http://localhost:9111/docs, site maps at http://localhost:9111/map.

Docker

docker compose up --build
# app on :9111, index persisted in the webdocs-data volume

MCP integration

Add to Cursor / VS Code / Claude Code MCP configuration:

{
  "webdocs": {
    "type": "http",
    "url": "http://localhost:9111/mcp"
  }
}

Exposed tools:

Tool	Purpose
search_docs	Hybrid search over every indexed chunk; returns text + source URLs
list_doc_pages	List indexed pages so the agent can pick one
get_doc_page	Full extracted text of one page

API reference

Endpoint	Description
POST /fetch_url	Start a crawl job ({url, max_pages?, max_depth?}); ?sync=true blocks until done
GET /job_progress	All jobs, or one via ?job_id=
GET /search_docs	?query=&top_k= hybrid search
GET /list_doc_pages	Every indexed page
GET /get_doc_page	?page_id= full page text
GET /map	Index of crawled sites (pure HTML, no JS)
GET /map/site/{root_id}	Hierarchical tree of one site
GET /map/page/{id}	Page view with breadcrumbs, siblings, children
GET /map/page/{id}/raw	Raw extracted text
POST /mcp	MCP JSON-RPC (initialize, tools/list, tools/call)
GET /health	Status + page/chunk counts

Site maps

Crawled pages keep parent/child relationships, so /map renders a real navigable tree per site — breadcrumbs from root, sibling navigation, children listing — in pure HTML (no JavaScript). Pages fetched individually from the same domain group under one root.

Configuration

All optional (see .env.example):

Variable	Default	Meaning
WEBDOCS_DB_PATH	data/webdocs.duckdb	Index location (falls back to in-memory if unwritable)
WEBDOCS_EMBEDDER	hashing	hashing (offline) or openai
OPENAI_API_KEY	—	Only for openai embedder
WEBDOCS_MAX_PAGES / WEBDOCS_MAX_DEPTH	50 / 3	Crawl limits
WEBDOCS_CHUNK_SIZE / WEBDOCS_CHUNK_OVERLAP	1200 / 150	Chunking

Testing

pytest tests/ -v      # 23 tests: crawler, chunker, embedder, DB, hybrid search, API, MCP

No test touches the network — the crawler tests run against an in-memory fake site injected through the fetcher interface, and each test gets a throwaway DuckDB file.

Roadmap

• [ ] Respect robots.txt and add per-domain rate limiting
• [ ] Incremental re-crawls (etag/last-modified) instead of full re-index
• [ ] Redis-backed worker pool using the existing compose service
• [ ] OCR/text extraction for linked PDFs
• [ ] DuckDB VSS extension (HNSW) once the index outgrows brute-force cosine

License

MIT — see LICENSE.