MCP Servers

Discover Awesome MCP Servers

Extend your agent with 84,516 capabilities via MCP servers.

All84,516

Fetch MCP Server

Okay, here's a breakdown of how you can fetch URLs from a webpage using Playwright, integrate it with an SSE (Server-Sent Events) MCP (Management Control Plane) server, and use Node.js with Express.js to orchestrate everything. I'll provide code snippets and explanations to guide you. **Conceptual Overview** 1. **Playwright (Web Scraping):** Playwright will be used to launch a browser, navigate to the target webpage, and extract the URLs you need. 2. **Node.js/Express.js (Server):** Express.js will create a web server that handles requests to start the scraping process and stream the results back to the client. 3. **SSE (Server-Sent Events):** SSE will be used to push the extracted URLs from the server to the client in real-time as they are found. This is more efficient than repeatedly polling the server. 4. **MCP (Management Control Plane):** The MCP part is a bit more abstract. It implies you have a system for managing and controlling the scraping process. This could involve: * Configuration: Specifying the target URL, selectors for extracting URLs, etc. * Monitoring: Tracking the progress of the scraping job. * Error Handling: Managing errors that occur during scraping. * Scaling: Distributing the scraping workload across multiple instances. **Code Example (Illustrative)** **1. Project Setup** ```bash mkdir playwright-sse-mcp cd playwright-sse-mcp npm init -y npm install playwright express eventsource ``` **2. `server.js` (Node.js/Express.js Server)** ```javascript const express = require('express'); const { chromium } = require('playwright'); const app = express(); const port = 3000; app.use(express.json()); // For parsing JSON request bodies // In-memory storage for SSE connections (replace with a more robust solution for production) const sseClients = []; app.get('/sse', (req, res) => { res.setHeader('Content-Type', 'text/event-stream'); res.setHeader('Cache-Control', 'no-cache'); res.setHeader('Connection', 'keep-alive'); res.flushHeaders(); const clientId = Date.now(); // Unique ID for the client const newClient = { id: clientId, res, }; sseClients.push(newClient); console.log(`${clientId} Connection open`); req.on('close', () => { console.log(`${clientId} Connection closed`); sseClients = sseClients.filter((client) => client.id !== clientId); }); }); function sendSSE(data) { sseClients.forEach((client) => { client.res.write(`data: ${JSON.stringify(data)}\n\n`); }); } app.post('/scrape', async (req, res) => { const { url, selector } = req.body; // Get URL and selector from request body if (!url || !selector) { return res.status(400).send('URL and selector are required.'); } console.log(`Starting scrape for ${url} with selector ${selector}`); try { const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto(url); const links = await page.locator(selector).evaluateAll(links => links.map(link => link.href)); for (const link of links) { sendSSE({ url: link }); // Send each URL via SSE } await browser.close(); console.log(`Scrape complete for ${url}`); res.status(200).send('Scrape started and URLs are being streamed.'); } catch (error) { console.error('Scrape error:', error); sendSSE({ error: error.message }); // Send error via SSE res.status(500).send('Scrape failed.'); } }); app.listen(port, () => { console.log(`Server listening at http://localhost:${port}`); }); ``` **3. `client.html` (Simple Client to Receive SSE)** ```html <!DOCTYPE html> <html> <head> <title>SSE Client</title> </head> <body> <h1>SSE Stream</h1> <ul id="urlList"></ul> <script> const urlList = document.getElementById('urlList'); const eventSource = new EventSource('http://localhost:3000/sse'); eventSource.onmessage = (event) => { const data = JSON.parse(event.data); if (data.url) { const listItem = document.createElement('li'); listItem.textContent = data.url; urlList.appendChild(listItem); } else if (data.error) { const listItem = document.createElement('li'); listItem.textContent = `Error: ${data.error}`; urlList.appendChild(listItem); } }; eventSource.onerror = (error) => { console.error('SSE error:', error); }; </script> </body> </html> ``` **4. Running the Example** 1. **Start the Server:** `node server.js` 2. **Open `client.html`** in your browser. 3. **Send a POST request to `/scrape`:** You can use `curl`, `Postman`, or a similar tool. For example: ```bash curl -X POST -H "Content-Type: application/json" -d '{"url": "https://www.example.com", "selector": "a"}' http://localhost:3000/scrape ``` Replace `"https://www.example.com"` with the URL you want to scrape and `"a"` with the CSS selector for the links you want to extract. **Explanation** * **`server.js`:** * Sets up an Express.js server. * `/sse` endpoint: Handles SSE connections. It sets the correct headers for SSE and keeps track of connected clients. * `sendSSE(data)`: Sends data to all connected SSE clients. * `/scrape` endpoint: * Receives the target URL and CSS selector from the request body. * Launches a Playwright browser. * Navigates to the URL. * Uses `page.locator(selector).evaluateAll()` to extract the `href` attributes of all elements matching the selector. * Sends each URL back to the client via SSE. * Handles errors and sends error messages via SSE. * **`client.html`:** * Creates an `EventSource` object to connect to the `/sse` endpoint. * Listens for `message` events from the server. * Parses the JSON data and displays the URLs in a list. * Handles errors. **Important Considerations for Production (MCP)** * **Configuration Management:** Instead of hardcoding the URL and selector in the `curl` command, you'd typically store them in a database or configuration file. Your MCP would provide an interface for managing these configurations. * **Job Queuing:** Use a message queue (e.g., RabbitMQ, Kafka) to queue scraping jobs. This allows you to handle a large number of requests without overloading the server. * **Scaling:** Run multiple instances of the scraping server behind a load balancer. The message queue will distribute the jobs across the instances. * **Monitoring:** Use a monitoring system (e.g., Prometheus, Grafana) to track the performance of the scraping servers, the number of jobs in the queue, and any errors that occur. * **Error Handling:** Implement robust error handling and retry mechanisms. For example, if a scraping job fails, you might retry it a few times before giving up. * **Rate Limiting:** Be respectful of the websites you are scraping. Implement rate limiting to avoid overloading their servers. * **Data Storage:** Instead of just displaying the URLs in the browser, you'll likely want to store them in a database or other data store. * **Authentication/Authorization:** Secure your MCP endpoints with authentication and authorization to prevent unauthorized access. * **Headless Mode:** Run the Playwright browser in headless mode (which is the default) for better performance. **Example of MCP Integration (Conceptual)** Let's say you have an MCP API endpoint `/api/scrape-jobs` that allows you to create new scraping jobs. The request body might look like this: ```json { "url": "https://www.example.com", "selector": "a", "callbackUrl": "https://your-data-store.com/api/store-data" // Where to send the scraped data } ``` Your server would then: 1. Receive the request to `/api/scrape-jobs`. 2. Validate the request. 3. Create a job in the message queue (e.g., RabbitMQ). 4. A worker process (one of your scraping server instances) would pick up the job from the queue. 5. The worker would scrape the URL, extract the data, and send it to the `callbackUrl`. 6. The worker would update the job status in the MCP (e.g., "in progress", "completed", "failed"). This is a simplified example, but it illustrates the basic principles of integrating Playwright with an SSE server and an MCP. The specific implementation will depend on your requirements and the architecture of your MCP. **Chinese Translation of Key Concepts** * **Web Scraping:** 网络爬虫 (wǎngluò páchóng) * **Server-Sent Events (SSE):** 服务器发送事件 (fúwùqì fāsòng shìjiàn) * **Management Control Plane (MCP):** 管理控制平面 (guǎnlǐ kòngzhì píngmiàn) * **Playwright:** Playwright (no direct translation, use the English name) * **Node.js:** Node.js (no direct translation, use the English name) * **Express.js:** Express.js (no direct translation, use the English name) * **URL:** 网址 (wǎngzhǐ) * **CSS Selector:** CSS 选择器 (CSS xuǎnzéqì) * **Endpoint:** 端点 (duāndiǎn) * **Message Queue:** 消息队列 (xiāoxi duìliè) * **Load Balancer:** 负载均衡器 (fùzài jūnhéngqì) * **Monitoring:** 监控 (jiānkòng) * **Error Handling:** 错误处理 (cuòwù chǔlǐ) * **Rate Limiting:** 速率限制 (sùlǜ xiànzhì) This comprehensive explanation and code example should give you a solid foundation for building your Playwright-based web scraping solution with SSE and MCP integration. Remember to adapt the code and architecture to your specific needs and environment. Good luck!

SQLx MCP Server

Provides comprehensive database management tools for PostgreSQL, MySQL, and SQLite databases. Enables querying table structures, executing read-only and write queries, exporting DDL statements, and managing database metadata through natural language.

shortcuts-mcp-server

youtube-context-mcp

A small MCP server that gives agents rich context about a YouTube video — its transcript, jump-to-the-moment deep links, metadata, and most-replayed moments — so they can answer questions, summarize, pull quotes, or surface highlights.

mcp-tw-company

An MCP server for querying Taiwan company registry open data, enabling search by name, number, directors, business items, and branch offices.

OpenAlex MCP Server

Enables AI agents to search and analyze OpenAlex scholarly database for OSINT research, including works, authors, institutions, funding, citations, and collaboration networks.

iris

Enables AI agents to create, list, update, and delete Microsoft 365 email drafts, with a hard guarantee that no message can be sent automatically, ensuring human review before any email goes out.

Valkey MCP Task Management Server

Enables AI agents to create, manage, and track tasks within plans using Valkey as the persistence layer. Supports plan and task management with Markdown notes, status tracking, and prioritization through multiple transport protocols (SSE, Streamable HTTP, STDIO).

nist-rag-mcp-server

A local FastMCP STDIO server exposing two tools (ask_nist_rag and get_nist_visual) and one read-only resource (nist://visuals/catalog) that provides grounded question answering over NIST AI RMF documents using local RAG with selective OCR, hybrid retrieval, and verified visual figure retrieval.

Pagos Data MCP Server

Enables Claude to retrieve BIN (Bank Identification Number) data for payment cards, with options for basic or enhanced insights through the Pagos API.

ts-mcp-durable-browser-automation

MCP server for robust browser automation of legacy web portals, featuring exactly-once execution via an SQLite-backed idempotency lock and resilient Playwright locators.

mcp_notes

MCP server for saving, reading, and listing Markdown notes in the filesystem. Enables agents to persistently store reports or notes as plain Markdown files.

Mcp Server Code Analyzer

Fiji MCP Server

Enables AI agents to control Fiji/ImageJ for microscopy image analysis through natural language commands, supporting operations like image opening, filtering, particle analysis, and automated workflows.

Librarian

Provides AI agents with persistent knowledge storage, enabling them to store, search, and retrieve text, documents, and files using semantic and keyword search via MCP tools.

Weather MCP Server

Provides current weather data and hourly forecasts for any location worldwide using the Open-Meteo API, accessible through HTTP transport without requiring an API key.

MCPGex

Enables LLMs to systematically develop and validate regex patterns by defining test cases with expected matches, testing patterns against them, and iteratively refining until all requirements are satisfied.

Mnemom

Provides trust infrastructure for AI agents by enabling reputation lookup, website trust scanning, and identity verification via MCP tools.

dpr-mcp

Provides a Model Context Protocol interface for secure, multi-agent collaboration on Git repositories, offering isolated workspaces, change requests, reviews, and approval workflows with Git as the source of truth.

MCPDemo - Visual SQL Chat Platform

An AI-driven data visualization and SQL chat platform that enables natural language to SQL conversion, interactive chart generation, and database querying through a comprehensive MCP interface. Supports multiple databases, intelligent chart recommendations, and secure data management with temporary link sharing.

Cypress Page Object Generator

An MCP server for automatically generating Cypress page objects from frontend components like Vue, React, or CoffeeScript.

app.wishpool/indonesia-payments-mcp

Enables AI agents to accept Indonesian payments including GoPay, QRIS, ShopeePay, DANA, bank transfers, and convenience store payments via Midtrans Snap's hosted checkout.

Google Search Console MCP Server

Enables AI agents to query Google Search Console data including search analytics, URL inspection, sitemap management, and site performance monitoring, with per-user OAuth authentication.

mcp-server-redmine

Enables AI assistants to create, update, and query Redmine issues and projects directly from conversations.

YouTube Knowledge MCP

Transforms YouTube into a queryable knowledge source with search, video details, transcript analysis, and AI-powered tools for summaries, learning paths, and knowledge graphs. Features quota-aware API access with caching and optional OpenAI/Anthropic integration for advanced content analysis.

Ariel

Ariel is an MCP server that exposes AT Protocol (Bluesky) operations as tools, enabling AI assistants to read, write, and interact on Bluesky with human-in-the-loop control.

PureContext MCP

Indexes codebases and lets AI agents retrieve precise code snippets (functions, classes, routes) instead of reading entire files, reducing token usage and improving accuracy.

Outlook MCP Server

Provides programmatic access to Microsoft Outlook mailboxes, enabling AI assistants to search, analyze, and extract insights from emails in personal and shared mailboxes.

x-mcp-server

Enables AI assistants to interact with X (Twitter) API v2 for posting tweets, searching, liking, retweeting, and more through natural language.

Google Health MCP

A local-first MCP server that enables AI agents to read user-authorized Google Health API v4 data from Fitbit, Pixel Watch, and partners via OAuth, with tokens never leaving the machine.