mcp-universal-crawler
Enables AI to autonomously search, filter, and download images from the web using natural language commands, with intelligent scoring and local storage.
README
🕷️ Universal Web Crawler MCP (SSE Version)
让 Claude 拥有“眼睛”和“手”，自动去互联网上搜索、筛选并下载图片。
这是一个基于 MCP (Model Context Protocol) 标准构建的通用图片爬虫工具。不同于普通的爬虫脚本，它是一个智能体工具——这意味着你不需要写代码，只需要用自然语言告诉 AI 你想要什么，它就会自动去办。
✨ 核心亮点
- 👀 全自动闭环：输入一句话（如“找点赛博朋克风格的壁纸”） -> AI 自动搜索 -> 智能评分筛选 -> 自动下载。
- 🧠 智能评分：内置评分系统，自动识别高质量来源（如优先爬取 .edu 或专业设计网站）。
- 🔌 SSE 架构：采用标准的 Server-Sent Events 协议，支持 HTTP 远程调用，部署更灵活。
- 🛡️ 隐私安全：代码开源，数据存储在本地，不经过第三方服务器。
🛠️ 快速开始
1. 环境准备
确保你的电脑上安装了：
2. 下载与安装
打开终端（Terminal / CMD），依次运行以下命令：
# 1. 克隆项目到本地
git clone [https://github.com/yywl07/mcp-universal-crawler.git](https://github.com/yywl07/mcp-universal-crawler.git)
cd mcp-universal-crawler
# 2. (推荐) 创建虚拟环境 - 防止污染你电脑的其他设置
python -m venv venv
# macOS/Linux 激活:
source venv/bin/activate
# Windows 激活:
venv\Scripts\activate
# 3. 安装项目依赖
pip install -r requirements.txt
Recommended Servers
playwright-mcp
A Model Context Protocol server that enables LLMs to interact with web pages through structured accessibility snapshots without requiring vision models or screenshots.
Magic Component Platform (MCP)
An AI-powered tool that generates modern UI components from natural language descriptions, integrating with popular IDEs to streamline UI development workflow.
Audiense Insights MCP Server
Enables interaction with Audiense Insights accounts via the Model Context Protocol, facilitating the extraction and analysis of marketing insights and audience data including demographics, behavior, and influencer engagement.
VeyraX MCP
Single MCP tool to connect all your favorite tools: Gmail, Calendar and 40 more.
graphlit-mcp-server
The Model Context Protocol (MCP) Server enables integration between MCP clients and the Graphlit service. Ingest anything from Slack to Gmail to podcast feeds, in addition to web crawling, into a Graphlit project - and then retrieve relevant contents from the MCP client.
Kagi MCP Server
An MCP server that integrates Kagi search capabilities with Claude AI, enabling Claude to perform real-time web searches when answering questions that require up-to-date information.
E2B
Using MCP to run code via e2b.
Neon Database
MCP server for interacting with Neon Management API and databases
Exa Search
A Model Context Protocol (MCP) server lets AI assistants like Claude use the Exa AI Search API for web searches. This setup allows AI models to get real-time web information in a safe and controlled way.
Qdrant Server
This repository is an example of how to create a MCP server for Qdrant, a vector search engine.