Yad2 Real Estate Scraper & Cloudflare Worker MCP Server 🏠🤖

An open-source, 100% free solution for scraping Yad2 real estate listings (apartments for rent and sale) and serving them through an MCP (Model Context Protocol) server hosted on Cloudflare Workers.

Designed for AI agents (Claude Desktop, Cursor, Gemini, ChatGPT Actions) to search, filter, and inspect apartments with direct links to original Yad2 notices.

---

⚡ Architecture & Bot Protection Strategy

┌───────────────────────────────────────┐
│        GitHub Actions (Weekly)        │
│  - Playwright Stealth + Chromium      │  ──(POST /ingest)──┐
│  - Evades Radware / ShieldSquare      │                    │
└───────────────────────────────────────┘                    ▼
                                              ┌──────────────────────────────┐
                                              │      Cloudflare Worker       │
                                              │                              │
┌───────────────────────────────────────┐     │  ┌────────────────────────┐  │
│        AI Agent / MCP Client          │ <==>│  │  MCP Server (JSON-RPC) │  │
│   (Claude, Cursor, Custom Agents)     │     │  └───────────┬────────────┘  │
└───────────────────────────────────────┘     │              │               │
                                              │              ▼               │
                                              │  ┌────────────────────────┐  │
                                              │  │ Cloudflare D1 Database │  │
                                              │  └────────────────────────┘  │
                                              └──────────────────────────────┘

1. Anti-Bot Bypass: Direct scraper requests from cloud datacenter IPs (like Cloudflare Workers) are blocked by Yad2's Radware / ShieldSquare protection. To overcome this on a 100% free tier, scraping is decoupled into a weekly GitHub Actions workflow using nodriver + xvfb-run (Virtual Framebuffer), which runs a real Chrome headful browser inside memory to bypass Radware JS challenges (stormcasterv2.js).
2. Edge Database (D1): Scraped listings are ingested into Cloudflare D1 (SQLite at the edge), providing sub-10ms lookup times and zero downtime.
3. MCP Server: Exposes MCP tools so AI agents can query apartments effortlessly.

---

🛠️ MCP Tools Offered

Tool	Description
search_apartments	Search apartments for rent or sale by city, neighborhood, street name, free text query, price range, room counts, square meters, and amenities. Returns listings with direct Yad2 links (url).
get_apartment_details	Get full details for a listing by ID, including images, full description, contact details, and direct Yad2 notice link.
get_scraping_status	View database stats (total counts for rent/sale) and the timestamp of the last weekly scrape run.

---

📊 Storage & Capacity Evaluation (Cloudflare D1 Free Tier)

Metric	Cloudflare D1 Free Tier Limit	Space Consumed by Yad2 Scraper	% of Free Quota Used
Storage Capacity	5 GB (5,000,000 KB)	~1.2 KB per apartment record (~12 MB for 10,000 listings)	0.24%
Daily Writes	100,000 writes / day	~1,000 - 10,000 batch upserts once a week	< 10% (on scrape day)
Daily Reads	5,000,000 reads / day	~10 - 100 queries / day by your AI agent	< 0.002%

💡 Conclusion: Storing 10,000 to 50,000 active apartment listings uses less than 1% to 1.2% of Cloudflare's free storage quota. It is 100% free and virtually impossible to exceed the tier limit for personal or community use!

---

🚀 Quick Setup & Deployment Guide

Prerequisites

• Node.js 18+ and npm
• Free Cloudflare Account
• Free GitHub Account

1. Clone & Install Dependencies

git clone https://github.com/your-username/yad2-mcp.git
cd yad2-mcp

# Install root dependencies
npm install

# Install scraper dependencies
cd scraper && npm install && cd ..

2. Create Cloudflare D1 Database

Log into Cloudflare via Wrangler and create your D1 SQLite database:

npx wrangler login
npx wrangler d1 create yad2-db

Copy the generated database_id from the output into your wrangler.toml:

[[d1_databases]]
binding = "DB"
database_name = "yad2-db"
database_id = "<YOUR_D1_DATABASE_ID>"

Initialize the database schema:

npm run d1:init:remote

3. Deploy Cloudflare Worker & Set Secret

Set an ingestion secret to secure the scraper upload endpoint:

npx wrangler secret put INGEST_SECRET
# Enter a secure key (e.g. my_super_secret_token_123)

Deploy the worker:

npm run deploy

Take note of your deployed Worker URL (e.g. https://yad2-mcp.<your-subdomain>.workers.dev).

---

📅 Setting Up Automated Scraper (GitHub Actions - Daily)

In your GitHub Repository, go to Settings > Secrets and variables > Actions and add two secrets:

1. WORKER_URL: https://yad2-mcp.<your-subdomain>.workers.dev
2. INGEST_SECRET: my_super_secret_token_123 (matching the Wrangler secret set above)

The GitHub Actions workflow (.github/workflows/scrape.yml) will automatically run every day at 02:00 UTC (0 2 * * *). You can also click Run workflow manually anytime in the GitHub Actions tab!

---

💻 Local Testing & Development

1. Test Cloudflare Worker Locally

# Initialize local D1 database
npm run d1:init:local

# Start Wrangler local dev server
npm run dev
# Worker runs at http://localhost:8787

2. Run Local Scraper Test

In another terminal:

cd scraper
WORKER_URL=http://localhost:8787 INGEST_SECRET=dev_ingest_secret_key npm run scrape

---

🤖 Connecting to AI Agents

Claude Desktop / Cursor Setup

Add the following to your claude_desktop_config.json or Cursor MCP settings:

{
  "mcpServers": {
    "yad2-apartments": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-fetch",
        "https://yad2-mcp.<your-subdomain>.workers.dev/mcp"
      ]
    }
  }
}

Or configure custom SSE connection directly to:

https://yad2-mcp.<your-subdomain>.workers.dev/mcp

---

⚠️ Disclaimer & Terms of Use

IMPORTANT: This project is strictly for educational, research, and technical evaluation purposes only. It is designed to demonstrate building Model Context Protocol (MCP) servers on Cloudflare Workers and integration with web automation workflows.

This tool should NOT be used for scraping Yad2 or any unauthorized automated data extraction in violation of Yad2's Terms of Service, Robots.txt, or applicable local data laws. The authors and contributors bear no responsibility for any misuse of this software.

---

📄 License

MIT License. Open source & free for community use!