Digest MCP Server

MCP server for web content digestion using browserless.io via puppeteer-core. Extracts fully rendered DOM content from dynamic web pages including SPAs and infinite scroll sites.

Features

• Connect to browserless.io cloud browsers
• Load web pages with configurable wait times
• Scroll down pages multiple times with delays
• Extract complete page content (HTML)

Installation

npm install
npm run build

Configuration

Set your browserless.io API key using one of these methods:

Option 1: Using .env file (recommended)

Create a .env file in the project root:

cp .env.example .env

Then edit .env and add your API key:

BROWSERLESS_API_KEY=your_api_key_here

Option 2: Using environment variable

export BROWSERLESS_API_KEY=your_api_key_here

Usage

Running the Server

The server uses stdio transport for MCP communication:

node build/index.js

Tool: web_content

Fetches web page content with optional scrolling and HTML cleanup.

Parameters:

• url (string, required): The URL to fetch
• initialWaitTime (number, optional): Time to wait in milliseconds after loading the page. Default: 3000
• scrolls (number, optional): Number of times to scroll down the page. Default: 5
• scrollWaitTime (number, optional): Time to wait in milliseconds between each scroll. Default: 1000
• cleanup (boolean, optional): Whether to clean up HTML (remove scripts, styles, SVG, forms, etc.) and keep only meaningful text content. Default: false

Returns:

• size (number): Size of the content in bytes
• content (string): The fetched HTML content

Example:

{
  "url": "https://example.com",
  "initialWaitTime": 2000,
  "scrolls": 3,
  "scrollWaitTime": 1000,
  "cleanup": true
}

How It Works

1. Connects to browserless.io using your API key via WebSocket
2. Creates a new page in the remote browser
3. Navigates to the specified URL (waits for DOM content loaded)
4. Waits 1 second for page stabilization
5. Waits for the initial wait time (default: 3 seconds)
6. Scrolls to the bottom of the page the specified number of times
7. After each scroll, intelligently waits for new content to load by:
   • Monitoring page height changes
   • Detecting dynamically loaded content
   • Waiting up to scrollWaitTime for new content (default: 3 seconds)
8. Waits for network to idle (AJAX requests complete)
9. Waits 1 additional second for JavaScript rendering
10. Returns the fully RENDERED DOM (not raw HTML source)
    • Includes all JavaScript-generated content
    • Includes all AJAX-loaded content
    • Includes all dynamically inserted elements
    • Uses document.documentElement.outerHTML for complete rendered state

Dynamic Content & Infinite Scroll

The tool is specifically designed for modern web applications with dynamic content:

AJAX/JavaScript Handling:

• ✅ Waits for network idle: Ensures all AJAX requests complete
• ✅ Returns rendered DOM: Gets actual content after JavaScript execution
• ✅ Not raw HTML source: Uses browser's rendered output
• ✅ Includes dynamic elements: Captures content inserted by React, Vue, Angular, etc.

Infinite Scroll Support:

• ✅ Scrolls to bottom: Triggers lazy-loading mechanisms
• ✅ Detects new content: Monitors page height changes
• ✅ Smart waiting: Exits early when content loads
• ✅ Multiple fallbacks: Keyboard scroll if JavaScript fails

Perfect for:

• Single Page Applications (React, Vue, Angular)
• Infinite scroll feeds (Twitter, Facebook, LinkedIn)
• Lazy-loaded images and content
• AJAX-powered content (search results, filters)
• Dynamic dashboards and admin panels

Tips for best results:

• Default scrolls: 5 works well for most pages with lazy-loaded content
• Increase scrolls to 10-15 for very long infinite scroll pages
• Set scrolls: 0 to disable scrolling for static pages
• Use scrollWaitTime of 1000-3000ms for slow-loading content (default: 1000ms)
• Increase initialWaitTime to 5000+ if page has heavy initialization
• For SPAs, allow time for initial JavaScript bootstrap
• Use cleanup: true to extract only meaningful text content without scripts, styles, and visual elements
• Use cleanup: false (default) to get the full rendered HTML

MCP Client Configuration

Add to your MCP client configuration (e.g., Claude Desktop):

{
  "mcpServers": {
    "digest": {
      "command": "node",
      "args": ["/path/to/digest-mcp/build/index.js"],
      "env": {
        "BROWSERLESS_API_KEY": "your_api_key_here"
      }
    }
  }
}

License

ISC