OmniMCP — Universal Web-to-MCP Gateway

Turn any website into a first-class MCP toolset in seconds.

OmniMCP is a universal MCP server that transforms any URL into a rich set of MCP tools. It can scrape content, perform deep crawls, run LLM-powered structured extraction, auto-detect and mount REST APIs via OpenAPI specs, and more — all exposed through a single SSE endpoint that any MCP client can connect to.

Philosophy

Modern AI agents are incredibly powerful at reasoning, but they are still fundamentally blind to the live web. Most agents either:

• Rely on brittle, pre-built scrapers for a handful of known sites, or
• Have no web access at all.

OmniMCP solves the "last mile" problem: any website → structured, tool-callable data without writing custom code for each domain.

It embodies three core principles:

• Universality — One server, any site.
• Zero-config intelligence — LLM extraction + automatic OpenAPI mounting when available.
• Production-grade reliability — Caching, retries, JS rendering, anti-bot hardening, and clean Markdown output.

Pain Points Solved

Pain Point	How OmniMCP Solves It
Agents can't access dynamic/JS-heavy sites	Playwright + crawl4ai with wait_for_js, user simulation, overlay removal
Every new site requires custom scraper code	scrape_url, crawl_site, smart_extract_page, css_extract_page work on any URL
REST APIs are invisible to agents unless manually integrated	connect_site auto-detects OpenAPI/Swagger specs and mounts them as native MCP tools
Repeated scraping wastes tokens and time	Built-in SQLite cache with configurable TTL (default 1h)
Structured data extraction is inconsistent	Optional LLM-based extraction (smart_extract_page) or deterministic CSS selectors
Agents need both raw content and precise data	Multiple extraction modes (Markdown, JSON schema, CSS, keyword search)

Use Cases

1. Research & Competitive Intelligence

# Agent wants latest pricing from a competitor
result = await scrape_url("https://competitor.com/pricing")

2. Automated Monitoring

Deep crawl a documentation site nightly and diff changes.

3. Dynamic API Integration

Point connect_site at a new SaaS product → instantly gain 20–80 new MCP tools from their OpenAPI spec.

4. Structured Data Extraction at Scale

Use smart_extract_page with natural language instructions + JSON schema to turn any page into typed data.

5. Internal Tooling for Agents

Give your agent a "web superpower" without maintaining dozens of site-specific scrapers.

Features

• 10 production tools (see table below)
• FastMCP 3.3.1 SSE transport (compatible with Claude Desktop, Cursor, Hermes, etc.)
• crawl4ai 0.8.6 backend with full JavaScript rendering
• Automatic OpenAPI mounting — turns REST APIs into MCP tools
• LLM-powered extraction (optional, via your local oMLX endpoint)
• SQLite result cache with namespace-based invalidation
• Proxy support for anti-bot sites
• Docker-ready

Tool Reference

Tool	Description	Best For
connect_site	Auto-detects OpenAPI spec or falls back to scrape mode	First contact with new domain
scrape_url	Clean Markdown + metadata from any URL	General browsing
crawl_site	Deep crawl with depth & page limits	Documentation, blogs, sites
smart_extract_page	Natural-language + optional JSON schema extraction via LLM	Structured data (pricing, products, etc.)
css_extract_page	Fast, deterministic CSS selector extraction	High-volume, stable layouts
search_page	Keyword search with surrounding context	Finding specific information
get_links	All internal + external hyperlinks	Graph building, discovery
fetch_api_endpoint	Raw HTTP client (GET/POST/PUT/DELETE)	Direct API calls
list_tools	Currently mounted tools	Self-inspection
clear_cache	Namespace or full cache flush	Forcing fresh data

Quick Start

# 1. Clone
git clone https://github.com/shagghiesuperstar/omni-mcp.git
cd omni-mcp

# 2. Create venv (Python ≥ 3.10 required)
python3 -m venv .venv
source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install browser
playwright install chromium

# 5. Run crawl4ai post-install
crawl4ai-setup

# 6. Configure (optional but recommended)
cp .env.example .env
# Edit .env and set LLM_BASE_URL if you want smart_extract_page to use your local LLM

# 7. Run
python server.py
# → SSE endpoint live at http://0.0.0.0:8055/sse

Hermes Agent Integration

Add to your config.yaml:

mcp_servers:
  omni_mcp:
    transport: sse
    url: http://localhost:8055/sse

Cross-machine (M4 ↔ M5 via Tailscale):

mcp_servers:
  omni_mcp:
    transport: sse
    url: https://your-m5-tailscale-hostname:8055/sse

Advanced Usage Examples

LLM-Powered Structured Extraction

result = await smart_extract_page(
    url="https://example.com/products",
    instruction="Extract all product names, prices, and availability as a list of objects",
    schema='{"type":"object","properties":{"products":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"string"},"in_stock":{"type":"boolean"}}}}}}'
)

Deep Crawl with Limits

pages = await crawl_site(
    base_url="https://docs.example.com",
    max_depth=3,
    max_pages=50
)

Auto-Mounting an API

await connect_site("https://api.example.com")
# → Instantly mounts all endpoints as MCP tools under the domain prefix

Opportunities & Future Directions

• Agent-native web layer — Become the default web ingestion layer for every major agent framework.
• Self-updating tool registry — Automatically discover and mount new APIs as sites publish OpenAPI specs.
• Multi-modal extraction — Combine LLM + vision for screenshot-based or PDF-heavy sites.
• Enterprise deployment — Add authentication, rate limiting, and audit logging for internal agent fleets.
• Community scrapers — Allow users to contribute high-quality CSS/LLM extraction templates.

License

MIT — use freely in personal and commercial agent projects.

---

Built with precision by Pi (Superstar_ops) • May 2026