DocMesh MCP

DocMesh indexes multiple public documentation sites into PostgreSQL and exposes cited passages through one MCP server. It is retrieval-only: it does not answer the technical question or mutate sources through MCP.

Architecture

The ingestion plane validates and DNS-pins public URLs, discovers pages from sitemaps or same-origin links, extracts article Markdown, chunks by headings, embeds changed chunks, and updates each document transactionally. The query plane routes a query to up to three sources, runs exact-symbol, PostgreSQL full-text, and pgvector searches, then combines ranks with reciprocal-rank fusion. See ARCHITECTURE.md.

Prerequisites and setup

• Node.js 22+ (Node 24 LTS works)
• pnpm 10+
• Docker, or PostgreSQL 17 with pgvector

pnpm install
cp .env.example .env
docker compose up -d
pnpm migrate

The checked-in migration uses vector(1536), matching .env.example. To change embedding dimensions safely, create a migration that rebuilds the vector column and HNSW index, set EMBEDDING_DIMENSION, then re-ingest all documents. Do not change only the environment variable.

Sources, ingestion, and search

pnpm cli source add --name AlphaDB --url https://docs.example.com --aliases alpha --tags database
pnpm cli source version:add --source alpha-db --version 2.0 --url https://docs.example.com/v2 --default
pnpm cli source list
pnpm cli ingest --source alpha-db
pnpm cli search --query "How does AlphaClient.connect() retry?"

For checked-in fixtures, no localhost crawl bypass is provided. They are loaded directly from trusted repository files:

pnpm fixtures:ingest
ALLOW_DEBUG=true pnpm cli search --query "Compare AlphaDB and BetaORM transactions" --debug
pnpm eval

MCP

Local stdio configuration:

{
  "mcpServers": {
    "docmesh": {
      "command": "pnpm",
      "args": ["--dir", "/absolute/path/to/docmesh-mcp", "mcp:stdio"],
      "env": { "DATABASE_URL": "postgres://docmesh:docmesh@localhost:5432/docmesh" }
    }
  }
}

Remote Streamable HTTP:

pnpm mcp:http
# endpoint: http://127.0.0.1:3000/mcp

The public surface is deliberately small: list_sources, search_docs, get_document, and docs://{sourceId}/{version}/{documentId} resources. Ingestion and arbitrary fetching are CLI-only.

Configuration

.env.example documents all variables. Core groups are database connection, embedding provider/model/dimension, optional OpenAI router and timeout, crawler limits, HTTP port, Playwright fallback, logging, and development debug output. Without an API key, fake deterministic embeddings and deterministic routing keep local tests credential-free. For useful production semantic retrieval, select the OpenAI embedding provider and re-ingest.

Playwright is installed as an optional package but browser binaries are not downloaded automatically. If rendering is enabled, run pnpm exec playwright install chromium explicitly.

Checks

pnpm format:check
pnpm lint
pnpm typecheck
pnpm test

Database integration tests are enabled with TEST_DATABASE_URL:

TEST_DATABASE_URL=postgres://docmesh:docmesh@localhost:5432/docmesh pnpm test

Security

Crawling arbitrary URLs is dangerous. DocMesh permits only HTTP(S), rejects credentials and private/loopback/link-local/multicast/reserved/metadata addresses, pins validated DNS answers for each request, revalidates every redirect, limits redirects/time/bytes/pages/depth, accepts textual content types, and restricts recursive links to configured origins and path prefixes. Playwright validates every browser request but browser-level DNS pinning depends on Chromium; keep it disabled for untrusted sources unless the deployment also enforces outbound network policy.

Known limitations

• HTML heuristics cover common documentation layouts, not every JavaScript framework.
• HTTP validators are stored, but conditional If-None-Match/If-Modified-Since requests are not yet issued; normalized hashes still prevent re-embedding unchanged content.
• Sequential fetching deliberately favors politeness and simplicity over large-crawl throughput.
• The router cache is process-local only through provider/client behavior; a durable routing cache is not included in the MVP.
• Remote deployment needs authentication, TLS, rate limiting, and network egress controls in front of the MCP endpoint.

Highest-value next improvements: conditional HTTP recrawls, a bounded concurrent crawl worker pool, and retrieval evaluation on a larger real documentation corpus.