web-scrapper-mcp

A portable MCP server that scrapes SPA/JS-rendered webpages and extracts structured API endpoint data. Registered globally in VS Code — callable from any project without reimplementing scraping.

Why this exists

Most API documentation sites (FMP, Tapetide, etc.) are React SPAs. httpx fetches only an empty shell. This server uses headless Chromium (Playwright) to fully render JavaScript, then extracts structured content.

Tech Stack

• Python 3.14 + uv
• Playwright Python (headless Chromium)
• FastMCP
• markdownify

Installation

git clone <repo>
cd web-scrapper-mcp
uv sync
uv run playwright install chromium
cp .env.example .env

Configuration

Edit .env:

SCRAPER_OUTPUT_DIR=output      # where screenshots are saved
SCRAPER_TIMEOUT_MS=30000       # page load timeout
# HTTPS_PROXY=http://proxy:8080  # optional — for Zscaler/corporate proxy

CLI Usage

# Generic scrape → markdown
uv run cli.py --url https://httpbin.org/html

# Scrape → plain text
uv run cli.py --url https://example.com --mode text

# Extract API endpoints (auto-detect site type)
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs --mode endpoints

# Save endpoint JSON to file
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs \
  --mode endpoints --out output/fmp_endpoints.json

# Screenshot
uv run cli.py --url https://example.com --mode screenshot

MCP Server Tools

scrape_page(url, wait_for?, selector?, mode?, wait_ms?)

Generic SPA scraper. Returns the rendered page as markdown, HTML, or plain text.

Param	Default	Description
url	required	URL to scrape
wait_for	networkidle	Playwright wait state
selector	""	CSS selector to wait for
mode	markdown	markdown | html | text
wait_ms	2000	Extra settle delay (ms)

Returns: {content, page_title, url, word_count, mode}

---

extract_api_endpoints(url, site_type?)

Extracts structured endpoint data. Auto-detects site type or accepts explicit override.

Site types: swagger | redoc | stoplight | fmp | generic

Returns:

{
  "endpoints": [
    {
      "method": "GET",
      "path": "/api/v3/stock/price",
      "summary": "Stock Price",
      "description": "...",
      "tags": ["Stock"],
      "parameters": [{"name": "symbol", "in": "query", "required": true, "type": "string"}]
    }
  ],
  "count": 247,
  "site_type": "fmp",
  "url": "..."
}

---

screenshot_page(url, full_page?)

Takes a full-page screenshot. Returns the saved file path and base64-encoded PNG for inline display.

Returns: {saved_path, base64}

---

scrape_and_seed(url, db_path, source_name, site_type?)

Power tool: scrapes API docs and seeds a SQLite database directly. The target DB must have an endpoints table (created automatically if absent).

Returns: {seeded, skipped, source, db_path}

Project Structure

web-scrapper-mcp/
  main.py               ← FastMCP server (4 tools)
  scraper.py            ← Core Playwright engine
  extractors/
    __init__.py
    generic.py          ← Any URL → markdown/html/text
    api_docs.py         ← Auto-detect swagger/redoc/stoplight/fmp/generic
    fmp.py              ← FMP developer docs specific extractor
  cli.py                ← CLI: uv run cli.py --url ...
  pyproject.toml
  .env.example
  .gitignore
  README.md
  .plan/                ← Architecture notes (gitignored)
  output/               ← Screenshots and output files (gitignored)