web-search-mcp

Best-in-class web search MCP server for grounded agent retrieval.

Features

• Multi-provider search (tavily, brave, exa, serper, searxng, duckduckgo)
• Provider fallback chain and optional ensembling
• Optional local semantic rerank (RERANKER=local) with @xenova/transformers
• SSRF-safe URL fetching with robots policy support
• Injection defense: <untrusted_content> wrapping, hidden-char stripping, heuristic detection
• HTML, PDF, RSS, and YouTube transcript extraction
• Reranking, dedupe/canonicalization, chunk anchors for citation
• Persistent search/page memory with SQLite (STATE_BACKEND=sqlite)
• MCP-native capabilities: tools, resources, prompts, completions, sampling, elicitation
• Evaluation harness (Recall@5, MRR@10) and provider A/B compare CLI
• Production endpoints: /livez, /readyz, /metrics, /health

Install

npm i
npm run build

Configure via .env

Copy and fill the template:

cp .env.example .env

Supported keys:

• SEARCH_PROVIDER=tavily|brave|exa|serper|duckduckgo
• TAVILY_API_KEY, BRAVE_API_KEY, EXA_API_KEY, SERPER_API_KEY
• SEARCH_PROVIDER_FALLBACK=brave,duckduckgo
• HOST=127.0.0.1
• STATE_BACKEND=memory|sqlite|redis
• SQLITE_PATH=.cache/web-search-mcp.sqlite
• DOMAIN_ALLOWLIST=*.gov,*.edu
• DOMAIN_BLOCKLIST=internal.example.com
• HTTP_RATE_LIMIT_RPS=30
• FETCH_MAX_BODY_BYTES=5000000
• PORT=8080
• MCP_BASE_PATH=/mcp (set /mcp/search if desired)
• CORS_ORIGIN=* (or a specific origin)
• MCP_AUTH_TOKEN=<secret> (optional bearer token for MCP endpoint)
• DEBUG=0|1

Run locally (default streamable MCP)

npm run dev

This auto-loads .env and starts a streamable MCP endpoint at:

• http://127.0.0.1:8080/mcp (or HOST/PORT + MCP_BASE_PATH from .env)

If you need stdio mode explicitly:

npm run dev:stdio

MCP client configuration

Streamable MCP (recommended)

Use URL-based config:

{
  "mcpServers": {
    "web-search": {
      "url": "http://127.0.0.1:8080/mcp"
    }
  }
}

If you set MCP_BASE_PATH=/mcp/search, use:

{
  "mcpServers": {
    "web-search": {
      "url": "http://127.0.0.1:8080/mcp/search"
    }
  }
}

If MCP_AUTH_TOKEN is enabled, include:

• Authorization: Bearer <token>

Stdio fallback (.cursor/mcp.json)

If your client requires command/stdio mode, use:

{
  "mcpServers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "web-search-mcp", "--stdio"],
      "env": {
        "SEARCH_PROVIDER": "tavily",
        "TAVILY_API_KEY": "YOUR_KEY",
        "SEARCH_PROVIDER_FALLBACK": "brave,duckduckgo"
      }
    }
  }
}

Claude Desktop (stdio fallback)

{
  "mcpServers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "web-search-mcp", "--stdio"],
      "env": {
        "SEARCH_PROVIDER": "tavily",
        "TAVILY_API_KEY": "YOUR_KEY"
      }
    }
  }
}

VS Code MCP (stdio fallback)

{
  "servers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "web-search-mcp", "--stdio"],
      "env": {
        "SEARCH_PROVIDER": "duckduckgo"
      }
    }
  }
}

Tools

• search
• fetch_url
• search_and_extract
• news_search
• academic_search
• get_youtube_transcript
• extract_pdf
• check_config

Prompts

• research_topic
• fact_check
• latest_news

Resources

• web-search://page/{id}
• web-search://history/{id}

CLI

npm run compare -- --query "model context protocol"
npm run eval

Inspector validation

npx @modelcontextprotocol/inspector npm run dev

Smoke test

curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/livez
curl -s http://127.0.0.1:8080/readyz
curl -s http://127.0.0.1:8080/metrics

Expected:

{ "status": "ok", "version": "0.1.0", "provider": "..." }

With auth enabled:

curl -i http://127.0.0.1:8080/mcp
# 401 unauthorized (without Bearer token)

Docker

Build:

docker build -t web-search-mcp .

Run with env file:

docker run --env-file .env -p 8080:8080 web-search-mcp

Then validate each tool:

• search
• fetch_url
• search_and_extract
• news_search
• academic_search
• get_youtube_transcript
• extract_pdf
• check_config

For nginx/Kubernetes deployment guidance see docs/NGINX.md and manifests in deploy/k8s plus Helm chart in charts/web-search-mcp.

Build and release

npm test
npm run build

Publish:

npm publish --provenance