MCP Documentation Crawler

A local, configurable Model Context Protocol server that crawls a documentation site, stores a local index, and gives Claude (or another MCP client) search and page-retrieval tools.

The default configuration targets GE Vernova Proficy Plant Applications 2025 documentation. It only follows links within the configured domain and path prefix.

Install and crawl

git clone https://github.com/codecracker2020/mcp-server-crawler.git
cd mcp-server-crawler
npm install
cp config.example.json config.json
npm run crawl -- --config=config.json
npm run search -- --config=config.json "production unit"

The generated local index is data/index.json (ignored by Git). Re-run npm run crawl to refresh it.

HTTP API for other applications

Start the API after crawling:

npm run serve -- --config=config.json

It listens on http://127.0.0.1:3100 by default and exposes:

curl "http://127.0.0.1:3100/health"
curl "http://127.0.0.1:3100/search?q=releasing%20a%20route&limit=5"
curl "http://127.0.0.1:3100/search?q=releasing%20a%20route&summarize=true"

GET /search returns JSON with the query, index timestamp, and matching url, title, excerpt, and score fields. Add summarize=true to include a Claude summary grounded only in those matching excerpts. This uses AWS Bedrock's normal credential chain (for example AWS_PROFILE, access keys, or an IAM role) and the configured bedrockRegion and bedrockModelId; it makes a billed Bedrock request. Use apiHost, apiPort, and apiCorsOrigin to configure the server. Keep the default loopback host unless another machine truly needs access; if you expose it on a network, set MCP_CRAWLER_API_KEY and send it as the X-API-Key request header.

Claude Desktop / Claude Code / VS Code configuration

Add this server configuration to the MCP settings used by your client, adjusting the absolute paths:

{
  "mcpServers": {
    "plant-applications-docs": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-server-crawler/src/server.js", "--config=/absolute/path/to/mcp-server-crawler/config.json"]
    }
  }
}

For Claude Code, the equivalent command is:

claude mcp add --transport stdio plant-applications-docs -- node /absolute/path/to/mcp-server-crawler/src/server.js --config=/absolute/path/to/mcp-server-crawler/config.json

After restarting or reloading the client, it exposes four tools: crawl_documentation, search_documentation, get_documentation_page, and crawler_status.

Configuration

Copy config.example.json to config.json and change any field. startUrl, allowedDomains, and urlPathPrefix form the crawl boundary. The server also accepts every major setting as an environment variable, which is useful for VS Code settings or CI:

JSON field	Environment variable
startUrl	MCP_CRAWLER_START_URL
allowedDomains	MCP_CRAWLER_ALLOWED_DOMAINS (comma-separated)
urlPathPrefix	MCP_CRAWLER_URL_PATH_PREFIX
maxPages	MCP_CRAWLER_MAX_PAGES
concurrency	MCP_CRAWLER_CONCURRENCY
requestDelayMs	MCP_CRAWLER_REQUEST_DELAY_MS
maxContentChars	MCP_CRAWLER_MAX_CONTENT_CHARS
dataDirectory	MCP_CRAWLER_DATA_DIR
apiHost	MCP_CRAWLER_API_HOST
apiPort	MCP_CRAWLER_API_PORT
apiCorsOrigin	MCP_CRAWLER_API_CORS_ORIGIN
apiKey	MCP_CRAWLER_API_KEY
bedrockRegion	MCP_CRAWLER_BEDROCK_REGION (or AWS_REGION)
bedrockModelId	MCP_CRAWLER_BEDROCK_MODEL_ID
bedrockMaxTokens	MCP_CRAWLER_BEDROCK_MAX_TOKENS

Use the configuration conservatively and comply with the documentation site's terms and robots policy. The crawler deliberately ignores non-HTML pages and removes navigation, scripts, styles, and other page chrome before indexing.