Site Knowledge

Crawls a website, indexes its content, and answers questions about it over MCP — with every passage carrying the URL and heading it came from.

Point it at a company's site. Ask "what's the refund window?" and get the answer along with the page and section it came from, so it can be checked.

---

The problem

"A chatbot that knows our website" is the most commonly requested AI feature a small business asks for, and the usual build has two failure modes that make it unusable in practice:

It can't cite anything. A confident paragraph with no source is worthless for anything a customer might act on — refund windows, pricing, eligibility. Nobody can verify it, and when it's wrong there's no way to tell.

The crawler eats the site. A naive link-follower walks into pagination, faceted filters, session URLs, and calendar pages, and either runs forever or indexes ten thousand near-identical pages.

This handles both, and returns retrieval quality signals rather than a flat list of hits.

How it works

site url ──▶ sitemap discovery ──▶ robots check ──▶ throttled fetch
                (else link crawl)                          │
                                                    extract main region
                                                           │
                                            chunk by heading, with overlap
                                                           │
                                                    BM25 index (JSON)
                                                           │
                                        MCP tools ◀──── search, cited

Design decisions

Sitemaps before link-following. A sitemap is the set of pages the owner wants indexed, so it sidesteps the traps a link crawler falls into. Discovery checks Sitemap: directives in robots.txt, then /sitemap.xml, then /sitemap_index.xml, recursing through sitemap indexes up to three levels. Link-crawling is the fallback, not the default.

Chunks are built from headings, not character offsets. Slicing at a fixed length puts unrelated topics in one passage and strands text from its heading. Here each passage inherits the heading it lived under, which is what makes "Refund policy" on /returns a citation rather than a guess. Sentence-boundary packing with a 30-word overlap keeps passages from cutting mid-sentence or losing context at the seam.

BM25, not embeddings — deliberately. No API key, no vector store, no per-query cost, and scores you can explain when a client asks why a passage came back. The semantic work belongs to the model consuming these tools: it reformulates the query and reads the passages. Embeddings are a real upgrade for synonym-heavy corpora and the obvious next step, but they cost the zero-setup property that makes this demoable in a minute. Headings and titles are weighted (3× and 2×) so a passage under a matching heading outranks one that mentions the term in passing.

Retrieval reports its own confidence. Every result carries which query terms matched and what fraction of them — so a passage matching 1 of 5 terms is labeled as such instead of presented as a hit. When all results are weak, the response says so explicitly. And when strong matches exist, weak ones are dropped rather than padding the list; single-keyword searches are never filtered, since there a 100%-coverage match of one term is exactly right.

robots.txt is enforced, not consulted. Same policy engine as price-monitor: longest-match precedence, */$ wildcards, agent-specific groups, Crawl-delay honored, per-origin throttling, honest self-identifying User-Agent, and no bot-detection evasion anywhere.

Extraction picks the largest content region, not the first. <main> when present, otherwise the biggest <article> — because taking the first <article> on a listing page that renders one per item throws the whole page away. Regions are only trusted if they hold ≥25% of the page's text; otherwise the full document is used with nav/header/footer/aside stripped.

MCP tools

claude mcp add site-knowledge -- node /path/to/site-knowledge/mcp-server.mjs

Tool	What it does
crawl_site	Crawl and index a site (sitemap or link-crawl, robots-respecting)
search_site	Retrieve relevant passages with source URL, heading, and match confidence
get_page	Full extracted text of one indexed page
list_pages	Everything the crawl captured, optionally filtered by path prefix
list_sites	All indexed sites with page and passage counts
fetch_page	Read a single URL without indexing it

search_site and list_pages take a pathPrefix to scope a query to a section (/docs, /support). Sites are addressed by host — example.com rather than the full origin — and a single indexed site is resolved automatically. Ambiguous references return the candidate list instead of picking one.

CLI

npm install

npm run crawl -- https://example.com --max 100
npm run search -- "refund policy" --site example.com --limit 5
npm run mcp
npm test

Indexes are written to data/<host>.json, one file per site.

Tests

npm test

38 tests on Node's built-in runner, no test dependency. Covers BM25 ranking and the coverage-floor rule, suffix normalisation, sitemap parsing (urlset, sitemapindex, CDATA, escaped entities), same-origin link filtering, heading-based chunking and its word floor, entity decoding, chrome stripping, and the largest-region extraction behavior — including a regression test for the listing-page <article> bug.

Stack

Node 20+ · native fetch · JSON index files · @modelcontextprotocol/sdk · zod. No vector database, no scraping framework, no headless browser.

Scope and limits

Reads public pages that robots.txt permits. It does not log in, bypass paywalls, or evade bot protection.

Client-rendered sites yield nothing. If content is assembled in the browser, a plain HTTP client sees an empty shell — crawl_site reports this rather than silently indexing nothing. Fixing it means a headless browser, which is a deliberate non-goal here.

Retrieval is lexical. A question phrased entirely in synonyms of the site's wording will miss; that's the tradeoff named above, and the response says when matches are weak rather than hiding it.

Check the terms of service of any site you crawl. Permitted by robots.txt is not the same as permitted by contract.