web-fetch-mcp

web-fetch-mcp

An MCP server that provides web scraping and Bing search capabilities, supporting both static and dynamic content through Puppeteer. It allows AI assistants to extract page summaries, SEO metadata, and full text with automatic detection for headless browser rendering.

Category
Visit Server

README

web-fetch-mcp

一个 Model Context Protocol (MCP) 服务器，为 AI 助手提供强大的网页抓取和搜索功能。

功能特性

  • 网页抓取：提取页面摘要、元数据、链接和文本内容
  • 动态页面支持：使用 Puppeteer 自动检测和渲染 SPA（单页应用）
  • SEO 元数据：提取 Open Graph、Twitter Card 和传统 Meta 标签
  • 网页搜索：内置 Bing 搜索集成
  • 智能检测：自动判断页面是否需要无头浏览器渲染

工具列表

工具 描述
fetch_page_summary 获取页面标题、链接和图片概览
fetch_page_metadata 提取 SEO 标签和 Open Graph 元数据
fetch_links 提取页面所有超链接
fetch_links_headless 使用无头浏览器提取链接（适用于动态页面）
fetch_page_text 提取页面主要文本内容
fetch_page_text_headless 使用无头浏览器提取文本（适用于动态页面）
search_web 使用 Bing 搜索引擎搜索

安装

使用 npm/npx（推荐）

# 全局安装
npm install -g @xiaozhuabcd1234/web-fetch-mcp

# 运行
npx web-fetch-mcp

使用 Bun

bun install @xiaozhuabcd1234/web-fetch-mcp

从源码运行

git clone https://github.com/xiaozhuabcd1234/web-fetch-mcp.git
cd web-fetch-mcp
bun install
bun run src/index.ts

使用方法

Claude Desktop 集成

添加到 claude_desktop_config.json：

{
  "mcpServers": {
    "web-fetch": {
      "command": "npx",
      "args": ["-y", "@xiaozhuabcd1234/web-fetch-mcp"]
    }
  }
}

项目结构

web-fetch-mcp/
├── src/
│   ├── index.ts           # MCP 服务器入口
│   ├── browser.ts         # Puppeteer 浏览器管理
│   ├── web-scraper.ts     # 核心抓取工具
│   ├── search.ts          # 网页搜索接口
│   └── search-engines/
│       └── bing.ts        # Bing 搜索实现

核心模块

  • browser.ts：管理 Puppeteer 浏览器实例，支持隐身模式和可配置选项
  • web-scraper.ts：使用 Cheerio 进行 HTML 解析，包含 SPA 检测启发式算法
  • search.ts：统一的搜索接口，支持多搜索引擎

环境要求

  • Bun 1.0+
  • Node.js 18+（备用方案）
  • Chromium/Chrome 浏览器（用于 Puppeteer）

环境变量

变量 描述 默认值
CHROME_PATH Chrome 可执行文件路径 自动检测

开源协议

GPL-3.0-only - 详见 LICENSE 文件。

Recommended Servers

playwright-mcp

playwright-mcp

A Model Context Protocol server that enables LLMs to interact with web pages through structured accessibility snapshots without requiring vision models or screenshots.

Official
Featured
TypeScript
Magic Component Platform (MCP)

Magic Component Platform (MCP)

An AI-powered tool that generates modern UI components from natural language descriptions, integrating with popular IDEs to streamline UI development workflow.

Official
Featured
Local
TypeScript
Audiense Insights MCP Server

Audiense Insights MCP Server

Enables interaction with Audiense Insights accounts via the Model Context Protocol, facilitating the extraction and analysis of marketing insights and audience data including demographics, behavior, and influencer engagement.

Official
Featured
Local
TypeScript
VeyraX MCP

VeyraX MCP

Single MCP tool to connect all your favorite tools: Gmail, Calendar and 40 more.

Official
Featured
Local
graphlit-mcp-server

graphlit-mcp-server

The Model Context Protocol (MCP) Server enables integration between MCP clients and the Graphlit service. Ingest anything from Slack to Gmail to podcast feeds, in addition to web crawling, into a Graphlit project - and then retrieve relevant contents from the MCP client.

Official
Featured
TypeScript
Kagi MCP Server

Kagi MCP Server

An MCP server that integrates Kagi search capabilities with Claude AI, enabling Claude to perform real-time web searches when answering questions that require up-to-date information.

Official
Featured
Python
E2B

E2B

Using MCP to run code via e2b.

Official
Featured
Neon Database

Neon Database

MCP server for interacting with Neon Management API and databases

Official
Featured
Exa Search

Exa Search

A Model Context Protocol (MCP) server lets AI assistants like Claude use the Exa AI Search API for web searches. This setup allows AI models to get real-time web information in a safe and controlled way.

Official
Featured
Qdrant Server

Qdrant Server

This repository is an example of how to create a MCP server for Qdrant, a vector search engine.

Official
Featured