<div align="center">

<img src="./assets/mywebsearch-logo.png" alt="WebSearch Forge logo" width="148" height="148">

WebSearch Forge MCP

A lightweight, unified WebSearch MCP for AI agents.

Search, read, crawl, parse, and research the public web through one MCP server.

Quick Start · Tools · Architecture · Configure Engines · 中文文档

</div>

What It Is

WebSearch Forge MCP is a self-contained MCP service for agent-driven web research:

Question → search sources → fetch pages → extract content → crawl related pages → compile evidence

It is split into clear layers:

• transports exposes MCP stdio and the optional FastAPI adapter.
• core handles configuration, caching, security checks, orchestration, and research workflows.
• providers implement search, fetching, extraction, crawling, and media features.
• SearXNG is the single search gateway for Bing, Baidu, Brave, DuckDuckGo, and other configured engines.

Highlights

One MCP server, six capabilities

Connect once to transports/mcp_stdio.py:

Tool	Purpose
search_web	Find candidate sources through SearXNG
fetch_web_content	Fetch and extract one public URL
search_and_fetch	Search, then read the top results
deep_research	Run multiple searches and compile a report
crawl_site	Recursively crawl a bounded same-domain site
youtube_transcript	Retrieve YouTube captions

Unified search gateway

The engines argument is passed to SearXNG as a filter. WebSearch Forge does not independently fan out to search websites:

WebSearch Forge MCP → SearXNG → Bing / Baidu / Brave / DuckDuckGo

Lightweight by default

MCP stdio needs no exposed application port or database server. The cache is SQLite. Optional packages add Trafilatura, Readability, stealth requests, Playwright, Office/PDF parsing, Scrapy, YouTube captions, and FastAPI without changing the MCP contract.

Bounded crawling

crawl_site supports a zero-dependency native backend and an optional Scrapy backend. It enforces page and depth limits, stays on the starting host, resolves relative links, and isolates page failures.

Quick Start

The following commands are for Windows PowerShell.

Install

cd D:\my-websearch\my_websearch
py -3.12 -m pip install -r requirements.txt

Start SearXNG

docker version
cd D:\my-websearch\my_websearch
docker compose up -d searxng
docker compose ps

The default gateway is http://127.0.0.1:8080. Configuration lives in config/searxng/settings.yml.

Configure the MCP client

Use mcp_config.example.json and keep an absolute path:

{
  "mcpServers": {
    "websearch-forge": {
      "command": "py",
      "args": [
        "-3.12",
        "D:\\my-websearch\\my_websearch\\transports\\mcp_stdio.py"
      ],
      "env": {
        "SEARXNG_URL": "http://127.0.0.1:8080",
        "CACHE_TTL": "300"
      }
    }
  }
}

The stdio adapter forces UTF-8 on Windows. Loopback traffic bypasses machine-wide proxy variables by default; set PROXY_URL explicitly when needed.

Tools

search_web

Search through SearXNG without downloading page bodies.

{
  "name": "search_web",
  "arguments": {
    "query": "Python 3.14 new features",
    "engines": ["bing", "baidu", "brave"],
    "limit": 5,
    "time_range": "month"
  }
}

fetch_web_content

Fetch and extract one public URL. HTML, PDF, DOCX, XLSX, PPTX, CSV, Markdown, and plain text are supported.

{
  "name": "fetch_web_content",
  "arguments": {
    "url": "https://www.python.org",
    "max_chars": 10000,
    "stealth_mode": "off",
    "render_mode": "auto",
    "extraction_mode": "auto"
  }
}

The response includes final URL, HTTP status, title, extraction method, word count, content, and discovered links.

search_and_fetch

Search first, then fetch the top results independently. A failed page is recorded on that item and does not cancel the batch.

{
  "name": "search_and_fetch",
  "arguments": {
    "query": "FastAPI MCP server",
    "limit": 3,
    "max_chars": 12000
  }
}

deep_research

Run related queries concurrently, fetch the strongest results, and return a Markdown report with source-level failures.

{
  "name": "deep_research",
  "arguments": {
    "queries": ["SearXNG engine configuration", "MCP stdio deployment"],
    "breadth": 3,
    "max_chars": 12000
  }
}

crawl_site

Crawl a same-host site with hard page and depth limits.

{
  "name": "crawl_site",
  "arguments": {
    "url": "https://www.python.org",
    "max_pages": 10,
    "max_depth": 2,
    "backend": "native",
    "stealth_mode": "off"
  }
}

native is the default. Install Scrapy and set backend to scrapy to use the optional backend. Each page reports URL, depth, status, fetch method, title, content, and word count.

youtube_transcript

Retrieve YouTube captions with optional source and translation languages.

Response Shape

{
  "query": "OpenAI",
  "provider": "searxng",
  "engines": ["bing", "baidu", "brave"],
  "total_results": 3,
  "results": [
    {
      "title": "OpenAI | Research & Deployment",
      "url": "https://openai.com/",
      "description": "...",
      "source": "openai.com",
      "engine": "bing",
      "score": 1.0
    }
  ],
  "partial_failures": []
}

Successful partial results are preserved. Engine, page, and document errors are returned as structured failure entries.

Architecture

flowchart LR
    A[Agent / MCP Client] -->|stdio JSON-RPC| B[transports/mcp_stdio.py]
    B --> C[core/service.py]
    C --> D[providers/search]
    D --> E[SearXNG]
    E --> F[Bing / Baidu / Brave / DDG]
    C --> G[providers/content]
    G --> H[HTTP / stealth / Playwright]
    C --> I[providers/crawl]
    C --> J[providers/media]
    C --> K[(SQLite TTL cache)]

• transports adapts protocols; MCP and FastAPI share the same service.
• core owns orchestration, cache policy, configuration, and URL security.
• providers/search talks to SearXNG and validates engine names.
• providers/content handles HTTP, stealth transport, rendering, and extraction.
• providers/crawl contains native and Scrapy crawling backends.
• providers/media contains the YouTube transcript provider.

Configure Search Engines

The project-owned SearXNG source is:

config/searxng/settings.yml

Two settings decide whether an engine can be called:

1. settings.yml enables the engine inside SearXNG.
2. providers/search/registry.py lists the accepted name in SUPPORTED_ENGINES.

Restart after changes:

cd D:\my-websearch\my_websearch
docker compose up -d --force-recreate searxng

If SearXNG does not provide the engine yet, implement that SearXNG engine first.

Security and Reliability

• Only HTTP and HTTPS URLs are accepted.
• Localhost, loopback, private IPv4, link-local, and private IPv6 targets are rejected.
• Redirect destinations are validated again.
• Search and fetch operations use a SQLite TTL cache, 300 seconds by default.
• Partial failures do not discard successful work.
• stdout is reserved for MCP JSON-RPC.

Optional Capabilities

Capability	Enablement
Trafilatura / Readability	Install requirements-api.txt
Stealth requests	Install curl-cffi and use stealth_mode=high
JavaScript rendering	Install Playwright and use render_mode=browser
PDF / DOCX / XLSX / PPTX	Install matching document packages
Scrapy crawling	Install Scrapy and use backend=scrapy
FastAPI HTTP service	Run py -3.12 -m transports.api
YouTube captions	Install youtube-transcript-api

Project Layout

my_websearch/
├── assets/                     # Project logo
├── config/searxng/             # SearXNG settings.yml
├── core/                       # Config, cache, security, orchestration
├── providers/
│   ├── search/                 # SearXNG gateway and engine allow-list
│   ├── content/                # Requests, rendering, extraction
│   ├── crawl/                  # Native and Scrapy backends
│   └── media/                  # YouTube transcript provider
├── transports/                 # MCP stdio and FastAPI adapters
├── tests/                      # Dependency-free self-checks
├── docker-compose.yml          # Local SearXNG gateway
├── mcp_config.example.json     # MCP client template
├── requirements.txt            # Dependency entry point
├── README.md                  # English documentation
└── README.zh-CN.md            # 中文文档

Development Check

cd D:\my-websearch
py -3.12 -m my_websearch.tests.test_server

Expected output:

my_websearch self-check: ok

Optional FastAPI service:

cd D:\my-websearch\my_websearch
py -3.12 -m transports.api

Then open http://127.0.0.1:8787/docs.

License

No license is imposed yet. Add a root-level LICENSE file before public distribution.