README

mcp-universal-crawler

MCP server that exposes Futurepedia crawling logic as tools for assistants.

What it does

Fetches a random AI tool from Futurepedia search API
Opens the tool detail page
Extracts structured fields (description, what is, key features, pros/cons, who uses, image)
Returns clean JSON for downstream assistant use

MCP tools

Futurepedia-specific:

futurepedia_random_tool() — get one random tool with parsed metadata
futurepedia_tools(count=3) — get several random tools in one call

Universal crawler:

crawl_url(url, timeout_sec=30) — parse generic webpage (text + title + links)
crawl_many(urls, timeout_sec=30) — batch crawl up to 20 URLs
crawl_sitemap(sitemap_url, limit=20, timeout_sec=30) — extract URLs from sitemap.xml
crawl_file(source, timeout_sec=30) — parse local/remote txt/md/html/pdf/docx
extract_structured(url, schema_json, timeout_sec=30) — CSS-selector extraction by schema

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e .
python -m universal_crawler_mcp.server

The server runs over stdio (default MCP mode).

Claude Desktop / MCP client config

{
  "mcpServers": {
    "universal-crawler": {
      "command": "python",
      "args": ["-m", "universal_crawler_mcp.server"],
      "cwd": "/absolute/path/to/mcp-universal-crawler"
    }
  }
}

Environment

Optional:

PROXY_URL — proxy URL used for HTTP requests

Notes

Crawler logic is adapted from litvan007/AI-slop-tg (src/futurepedia.py) and wrapped for MCP usage.

mcp-universal-crawler

README

mcp-universal-crawler

What it does

MCP tools

Quick start

Claude Desktop / MCP client config

Environment

Notes

Recommended Servers