WebSearchAndCrawl

An MCP Server for Authenticated Web Crawling, Searching, and Document Processing

---.

🚀 Purpose

WebSearchAndCrawl is an MCP (Model Context Protocol) server designed to:

1. Crawl websites (including authenticated ones) using Firefox session tokens or browser automation.
2. Search crawled content for regex matches and store results in a structured index.
3. Download and parse documents (PDF, DOCX, XLSX, etc.) from crawled sites.
4. Stream results in real-time via HTTP for integration with MCP clients.
5. Respect robots.txt and enforce rate limits (5 requests/sec, 5 threads max).

This tool is ideal for:

• Researchers who need to scrape authenticated or dynamic websites.
• Developers building AI agents that require web data.
• Automation of repetitive web tasks (e.g., monitoring, data extraction).

---

🔧 Features

Feature	Description
Authenticated Crawling	Uses Firefox session tokens to access logged-in pages.
Browser Automation	Falls back to Playwright for dynamic content or login forms.
Domain Whitelisting	Only crawls URLs matching a comma-delimited list of domains.
Depth-limited Crawling	Configurable crawl depth (1-9 layers).
Regex Search	Search crawled content or index for regex patterns.
Document Parsing	Extracts text from PDF, DOCX, XLSX, and TXT files.
Real-time Streaming	Results are streamed as JSONL (chunked by page).
Indexing	Stores results in JSON files per domain for later search.
Rate Limiting	Enforces 5 requests/sec and 5 threads max.
Resume Support	Can resume interrupted crawls from checkpoints.
Session Validation	Validates token scopes to prevent misuse.

---

📦 Installation

Prerequisites

1. Python 3.9+ (recommended: 3.11+).
2. Firefox (required for browser automation).
3. System Libraries (for document parsing):
   • PDF: poppler-utils (Linux) or pdfminer.six (cross-platform).
   • DOCX/XLSX: python-docx, openpyxl.

Steps

1. Clone the Repository

git clone https://github.com/bpweatherill/WebSearchAndCrawl.git
cd WebSearchAndCrawl

2. Set Up a Virtual Environment

python -m venv venv
source venv/bin/activate  # Linux/Mac
# OR
venv\Scripts\activate   # Windows

3. Install Dependencies

pip install -r requirements.txt

4. Install Playwright Browsers

playwright install firefox

5. (Optional) Configure Environment Variables

Create a .env file in the project root:

# Server
MCP_PORT=8808
MCP_HOST=0.0.0.0

# Crawler
MAX_DEPTH=9
MAX_THREADS=5
RATE_LIMIT=5
REQUEST_TIMEOUT=10
MAX_MEMORY_MB=1024

# Firefox
FIREFOX_PROFILE=my_profile  # Optional: Specific Firefox profile
DEFAULT_SEARCH_ENGINE=google

# Directories
INDEX_DIR=./index
DOWNLOADS_DIR=./downloads
CHECKPOINTS_DIR=./checkpoints

---

🏃 Usage

1. Start the MCP Server

python -m server.main

The server will start on http://localhost:8808 (or the port specified in .env).

2. MCP Tools (HTTP Endpoints)

All tools return JSON responses and support streaming for real-time results.

Endpoint	Method	Description	Request Body
/crawl_website	POST	Crawl a website and stream results.	CrawlRequest
/search_index	POST	Search the local index for regex matches.	SearchIndexRequest
/download_documents	POST	Download documents matching a regex.	DownloadRequest
/get_search_results	POST	Use Firefox's search engine to fetch results.	WebSearchRequest
/list_indexed_domains	GET	List all domains with indexed content.	-
/health	GET	Health check.	-

---

Request/Response Schemas

CrawlRequest

{
  "url": "https://www.nasa.gov",
  "whitelist_domains": "nasa.gov",
  "max_depth": 3,
  "use_token": false,
  "firefox_profile": "my_profile"
}

• url: Starting URL for the crawl.
• whitelist_domains: Comma-delimited list of allowed domains (e.g., "nasa.gov,spacex.com").
• max_depth: Maximum crawl depth (1-9).
• use_token: Use Firefox session token if available.
• firefox_profile: Firefox profile name (optional).

Streamed Response (JSONL):

{
  "excerpt": "NASA's Perseverance Rover lands on Mars...",
  "full_text": "Full article text here...",
  "url": "https://www.nasa.gov/mars2020",
  "timestamp": "2024-05-20T12:00:00Z",
  "domain": "nasa.gov"
}

---.

SearchIndexRequest

{
  "domain": "nasa.gov",
  "regex": ".*Mars.*",
  "max_results": 10
}

• domain: Domain to search (e.g., "nasa.gov").
• regex: Regex pattern to match.
• max_results: Maximum number of results to return.

Response:

[
  {
    "url": "https://www.nasa.gov/mars2020",
    "excerpt": "NASA's Perseverance Rover lands on Mars...",
    "timestamp": "2024-05-20T12:00:00Z"
  }
]

---.

DownloadRequest

{
  "domain": "nasa.gov",
  "regex": ".*\\.pdf$",
  "output_dir": "./downloads/nasa.gov"
}

• domain: Domain to download from.
• regex: Regex pattern for files to download (e.g., "*.pdf").
• output_dir: Custom output directory (optional).

Streamed Response (JSONL):

{
  "filename": "./downloads/nasa.gov/mars_rover.pdf",
  "url": "https://www.nasa.gov/pdf/mars_rover.pdf",
  "parsed_text": "Extracted text from PDF..."
}

---.

WebSearchRequest

{
  "query": "NASA Mars missions",
  "search_engine": "google",
  "use_token": false,
  "firefox_profile": "my_profile"
}

• query: Search query.
• search_engine: Search engine (default: Firefox default).
• use_token: Use Firefox session token if available.
• firefox_profile: Firefox profile name (optional).

Streamed Response (JSONL):

{
  "title": "Mars 2020 Mission - NASA",
  "url": "https://www.nasa.gov/mars2020",
  "snippet": "Learn about the Perseverance Rover..."
}

---

🔍 Examples

1. Crawl NASA.gov and Index Results

curl -X POST http://localhost:8808/crawl_website \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.nasa.gov",
    "whitelist_domains": "nasa.gov",
    "max_depth": 2,
    "use_token": false
  }'

2. Search Indexed Content for "Mars"

curl -X POST http://localhost:8808/search_index \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "nasa.gov",
    "regex": ".*Mars.*",
    "max_results": 5
  }'

3. Download PDFs from NASA.gov

curl -X POST http://localhost:8808/download_documents \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "nasa.gov",
    "regex": ".*\\.pdf$"
  }'

4. Use Firefox to Search Google

curl -X POST http://localhost:8808/get_search_results \
  -H "Content-Type: application/json" \
  -d '{
    "query": "NASA Mars missions",
    "search_engine": "google"
  }'

---

📁 Project Structure

WebSearchAndCrawl/
│
├── server/                          # Core server logic
│   ├── __init__.py
│   ├── main.py                       # FastAPI app + MCP tools
│   ├── config.py                     # Configuration settings
│   ├── schemas.py                    # Pydantic request/response models
│   │
│   ├── firefox/                      # Firefox browser automation
│   │   ├── __init__.py
│   │   ├── controller.py              # Playwright Firefox management
│   │   └── token_manager.py           # Session token handling
│   │
│   ├── crawler/                     # Web crawling logic
│   │   ├── __init__.py
│   │   ├── crawler.py                # Main crawling logic
│   │   └── rate_limiter.py            # Thread/rate limiting
│   │
│   ├── indexer/                     # Indexing and search
│   │   ├── __init__.py
│   │   ├── indexer.py                 # JSON index management
│   │   └── search_engine.py           # Regex search
│   │
│   ├── downloader/                  # Document downloading and parsing
│   │   ├── __init__.py
│   │   ├── downloader.py              # Download logic
│   │   └── parsers/                  # File type parsers
│   │       ├── __init__.py
│   │       ├── pdf_parser.py
│   │       ├── docx_parser.py
│   │       └── xlsx_parser.py
│   │
│   └── streamer.py                   # Chunked JSON streaming
│
├── tests/                           # Unit and integration tests
│   ├── __init__.py
│   ├── test_firefox.py
│   └── test_crawler.py
│
├── index/                           # Index files (auto-generated)
│   ├── nasa.gov.json
│   └── ...
│
├── downloads/                       # Downloaded documents (auto-generated)
│   ├── nasa.gov/
│   │   ├── document1.pdf
│   │   └── ...
│   └── ...
│
├── checkpoints/                     # Crawl checkpoints (auto-generated)
│   └── ...
│
├── requirements.txt                 # Python dependencies
├── .env.example                     # Example environment variables
└── README.md                        # This file

---

⚙️ Configuration

Environment Variables

Variable	Default	Description
MCP_PORT	8808	HTTP server port.
MCP_HOST	0.0.0.0	HTTP server host.
MAX_DEPTH	9	Maximum crawl depth (1-9).
MAX_THREADS	5	Maximum concurrent threads.
RATE_LIMIT	5	Maximum requests per second.
REQUEST_TIMEOUT	10	Timeout for requests (seconds).
MAX_MEMORY_MB	1024	Maximum memory usage (MB).
FIREFOX_PROFILE	None	Firefox profile name (optional).
DEFAULT_SEARCH_ENGINE	google	Default search engine.
INDEX_DIR	./index	Directory for index files.
DOWNLOADS_DIR	./downloads	Directory for downloaded files.
CHECKPOINTS_DIR	./checkpoints	Directory for crawl checkpoints.

---

🛡️ Security Considerations

1. Session Tokens:

   • Tokens are only stored in memory (not persisted to disk).
   • Token scopes are validated to prevent misuse (e.g., a token for nasa.gov cannot be used for evil.com).

2. Input Sanitization:

   • All inputs (URLs, regex, etc.) are sanitized to prevent injection attacks.

3. Rate Limiting:

   • Enforces 5 requests/sec and 5 threads max to avoid overwhelming servers.

4. robots.txt Compliance:

   • The crawler respects robots.txt and skips disallowed URLs.

5. Whitelisting:

   • Only URLs matching the whitelisted domains are crawled.

---

🚀 Enhancements (Roadmap)

Enhancement	Description	Priority
Persistent Tokens	Store tokens in an encrypted file for persistence across restarts.	Medium
Full robots.txt Parsing	Properly parse robots.txt rules instead of simple checks.	Medium
Advanced Pagination Handling	Detect and follow pagination links (e.g., "Next" buttons).	High
Lazy-Loading Support	Detect and trigger lazy-loaded content (e.g., infinite scroll).	High
Checkpointing	Save crawl state to resume interrupted crawls.	High
Full-Text Search	Support full-text search in addition to regex.	Low
Database Backend	Replace JSON files with SQLite/PostgreSQL for scalability.	Low
Distributed Crawling	Support horizontal scaling with multiple workers.	Low
Docker Support	Add a Dockerfile for containerized deployment.	Medium
Authentication Helpers	Built-in support for common auth methods (OAuth, SAML).	Medium
Proxy Support	Add proxy support for crawling behind firewalls.	Low
Custom Headers	Allow users to specify custom headers for requests.	Medium
Webhook Notifications	Notify a webhook URL when new results are found.	Low

---

🤝 Contributing

1. Fork the repository.
2. Create a feature branch (git checkout -b feature/your-feature).
3. Commit your changes (git commit -m "Add your feature").
4. Push to the branch (git push origin feature/your-feature).
5. Open a Pull Request.

---

📜 License

This project is licensed under the MIT License. See LICENSE for details.

---

📞 Support

• Issues: Report bugs or request features in the GitHub Issues tab.
• Discussions: Join the GitHub Discussions for Q&A.

---

🏆 Acknowledgments

• Playwright: For browser automation.
• FastAPI: For the HTTP server.
• pdfminer.six: For PDF parsing.
• python-docx/openpyxl: For Office file parsing.