Claude Web Scraper MCP

Claude Web Scraper MCP

A simple MCP server that integrates eGet web scraper with Claude for Desktop. This connector allows Claude to scrape web content through your local eGet API, enabling search, summarization, and analysis of websites directly in conversations.

vishwajeetdabholkar

Developer Tools
Visit Server

README

Claude Web Scraper MCP

A simple Model Context Protocol (MCP) server that connects Claude for Desktop to a locally running eGet web scraper. This allows Claude to scrape website content through your local API.

Prerequisites

  • Claude for Desktop
  • Python 3.7+
  • eGet web scraper (from https://github.com/vishwajeetdabholkar/eGet-Crawler-for-ai)

Setup Instructions

1. Set up eGet Web Scraper

First, make sure you have the eGet web scraper running:

# Clone the eGet repository
git clone https://github.com/vishwajeetdabholkar/eGet-Crawler-for-ai
cd eGet-Crawler-for-ai

# Set up and run eGet according to its instructions
# (typically using Docker or local Python installation)

# Verify the API is running (default: http://localhost:8000/api/v1/scrape)

2. Set up the MCP Server

# Create project directory
mkdir claude-scraper-mcp
cd claude-scraper-mcp

# Set up UV and virtual environment
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
uv add "mcp[cli]" httpx

# Create the MCP server script
touch scrape_mcp_server.py

Copy the scrape_mcp_server.py code into the file.

3. Configure Claude for Desktop

  1. Create or edit the Claude desktop configuration:
# On macOS
mkdir -p ~/Library/Application\ Support/Claude/
  1. Add this configuration to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
    "mcpServers": {
        "scrape-service": {
            "command": "/absolute/path/to/claude-scraper-mcp/.venv/bin/python",
            "args": [
                "/absolute/path/to/claude-scraper-mcp/scrape_mcp_server.py"
            ]
        }
    }
}

Replace the paths with the actual absolute paths to your virtual environment and script.

  1. Restart Claude for Desktop

Usage

Once set up, you can use Claude to scrape websites with commands like:

  • "Scrape the content from https://example.com and summarize it"
  • "Get information about the website at https://news.ycombinator.com"

Troubleshooting

If you encounter issues:

  1. Check that eGet scraper is running
  2. Verify the API endpoint in the script matches your eGet configuration
  3. Make sure Claude for Desktop is using the correct Python interpreter
  4. Restart Claude for Desktop after making changes to the configuration

Recommended Servers

playwright-mcp

playwright-mcp

A Model Context Protocol server that enables LLMs to interact with web pages through structured accessibility snapshots without requiring vision models or screenshots.

Official
Featured
TypeScript
Magic Component Platform (MCP)

Magic Component Platform (MCP)

An AI-powered tool that generates modern UI components from natural language descriptions, integrating with popular IDEs to streamline UI development workflow.

Official
Featured
Local
TypeScript
MCP Package Docs Server

MCP Package Docs Server

Facilitates LLMs to efficiently access and fetch structured documentation for packages in Go, Python, and NPM, enhancing software development with multi-language support and performance optimization.

Featured
Local
TypeScript
Claude Code MCP

Claude Code MCP

An implementation of Claude Code as a Model Context Protocol server that enables using Claude's software engineering capabilities (code generation, editing, reviewing, and file operations) through the standardized MCP interface.

Featured
Local
JavaScript
@kazuph/mcp-taskmanager

@kazuph/mcp-taskmanager

Model Context Protocol server for Task Management. This allows Claude Desktop (or any MCP client) to manage and execute tasks in a queue-based system.

Featured
Local
JavaScript
Linear MCP Server

Linear MCP Server

Enables interaction with Linear's API for managing issues, teams, and projects programmatically through the Model Context Protocol.

Featured
JavaScript
mermaid-mcp-server

mermaid-mcp-server

A Model Context Protocol (MCP) server that converts Mermaid diagrams to PNG images.

Featured
JavaScript
Jira-Context-MCP

Jira-Context-MCP

MCP server to provide Jira Tickets information to AI coding agents like Cursor

Featured
TypeScript
Linear MCP Server

Linear MCP Server

A Model Context Protocol server that integrates with Linear's issue tracking system, allowing LLMs to create, update, search, and comment on Linear issues through natural language interactions.

Featured
JavaScript
Sequential Thinking MCP Server

Sequential Thinking MCP Server

This server facilitates structured problem-solving by breaking down complex issues into sequential steps, supporting revisions, and enabling multiple solution paths through full MCP integration.

Featured
Python