deepcrawl-mcp

<p align="center"> <img src="static/banner.svg" alt="deepcrawl" width="700" /> </p>

<p align="center"> <a href="https://www.npmjs.com/package/deepcrawl-mcp"><img src="https://img.shields.io/npm/v/deepcrawl-mcp.svg?style=flat-square&color=06b6d4" alt="npm version" /></a> <img src="https://img.shields.io/badge/tools-4-8b5cf6?style=flat-square" alt="4 tools" /> <img src="https://img.shields.io/badge/price-free-06b6d4?style=flat-square" alt="free" /> <img src="https://img.shields.io/badge/API_keys-none-06b6d4?style=flat-square" alt="no API keys" /> </p>

<p align="center"> <a href="cursor://anysphere.cursor-deeplink/mcp/install?name=deepcrawl-mcp&config=eyJ0eXBlIjoic3RkaW8iLCJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsImRlZXBjcmF3bC1tY3BAbGF0ZXN0Il19"><img src="https://img.shields.io/badge/Install_in-Cursor-blue?style=for-the-badge&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZD0iTTEyIDJMMiAxMmwxMCAxMCAxMC0xMEwxMiAyeiIgZmlsbD0id2hpdGUiLz48L3N2Zz4=" alt="Install in Cursor" /></a> <a href="https://soflution1.github.io/BrandCheck/install.html?name=deepcrawl-mcp&pkg=deepcrawl-mcp"><img src="https://img.shields.io/badge/Install_via-Web-green?style=for-the-badge" alt="Install via Web" /></a> </p>

---

Clone, scrape and crawl any website. Free Firecrawl alternative. No API keys, no rate limits, no subscription.

Quick Start

Cursor (one-click): Click the "Install in Cursor" button above.

Manual MCP config:

{
  "mcpServers": {
    "deepcrawl-mcp": {
      "command": "npx",
      "args": ["-y", "deepcrawl-mcp@latest"]
    }
  }
}

CLI:

npx deepcrawl-mcp@latest

Tools

deepcrawl_scrape

Scrape a single page and return clean markdown. Extracts title, description, links, images, and metadata. Strips navigation, footer, ads, and tracking.

"Scrape https://example.com and give me the main content"

Parameter	Default	Description
url	required	Page URL to scrape
mainContentOnly	true	Extract only main content (skip nav/footer)
includeLinks	true	Include discovered links
includeImages	true	Include image URLs

deepcrawl_clone

Clone a full page with all assets: HTML, CSS, JS, images, fonts, favicons. Downloads everything into a local folder, rewrites URLs to relative paths. Open index.html in a browser and it works.

"Clone https://competitor.com into a local folder"

Parameter	Default	Description
url	required	Page URL to clone
outputDir	~/deepcrawl-clones/<domain>	Output folder
depth	0	Link depth: 0 = single page, 1+ = follow links

deepcrawl_crawl

Crawl an entire site following internal links. Returns every page as clean markdown. Great for content analysis, SEO audit, or feeding a RAG pipeline.

"Crawl https://docs.example.com and return all pages as markdown"

Parameter	Default	Description
url	required	Starting URL
maxPages	20	Max pages to crawl (max: 100)
includeImages	false	Include image URLs per page

deepcrawl_map

Discover all URLs from a site via sitemap.xml parsing and homepage link crawling. Run this before a crawl to see the site's scope.

"Map all pages on https://example.com"

Parameter	Default	Description
url	required	Site URL
maxUrls	200	Max URLs to discover

vs Firecrawl

	deepcrawl	Firecrawl
Price	Free	$19+/mo
API key	None	Required
Rate limits	None	Yes
Scrape to markdown	Yes	Yes
Full site crawl	Yes	Yes
Site map	Yes	Yes
Clone with assets	Yes	No
JS rendering	Yes (via Playwright)	Yes
Anti-bot bypass	Partial (UA rotation, headers, delays)	Yes

deepcrawl handles static sites out of the box. For JS-heavy SPAs (React, Next.js, SvelteKit), install Playwright for full rendering:

npm install -g playwright
npx playwright install chromium

Once installed, deepcrawl auto-detects Playwright and enables JS rendering. Use jsRender: true on any tool to activate it. All tools also include UA rotation, realistic browser headers, and random delays to avoid basic bot detection.

Also by Soflution

• brandcheck - Check brand name availability across 27 platforms
• depsonar - Dependency audit, security scan, license check

License

MIT