fetchaller-mcp

Fetch any website in Claude Code without permission prompts. Built-in web search, Reddit support, and automatic bot challenge bypass.

Why fetchaller?

Claude Code's built-in WebFetch asks permission for every new domain and blocks Reddit entirely. fetchaller fixes both:

• fetch: Read any URL — automatically bypasses Cloudflare, Akamai, and other bot challenges
• search: Web search via Google + DuckDuckGo
• browse_reddit: Browse subreddit listings (hot/new/top/rising)
• search_reddit: Search Reddit posts globally or within a subreddit
• search_marketplace: Search Kijiji, Craigslist, and Facebook Marketplace simultaneously with human-readable params (city name, category, price range)
• search_realtor: Search Canadian homes on realtor.ca for sale or rent with full filters (location, price, beds, baths, property/building type, ownership)
• get_aliexpress_product: AliExpress product details (price, specs, ratings, reviews)
• search_aliexpress: Search AliExpress products with price filters and sorting
• get_alibaba_product: Alibaba.com B2B product details (tiered pricing, MOQ, lead times, supplier info)
• search_alibaba: Search Alibaba.com B2B products

Quick Start

Local Installation (stdio mode)

# Clone and install
git clone https://github.com/Averyy/fetchaller-mcp.git
cd fetchaller-mcp
uv sync && patchright install chromium

# Add to Claude Code
claude mcp add fetchaller -- $(pwd)/.venv/bin/python -m fetchaller.main

Add permissions to ~/.claude/settings.json:

{
  "permissions": {
    "allow": [
      "mcp__fetchaller__fetch",
      "mcp__fetchaller__search",
      "mcp__fetchaller__browse_reddit",
      "mcp__fetchaller__search_reddit",
      "mcp__fetchaller__search_marketplace",
      "mcp__fetchaller__search_realtor",
      "mcp__fetchaller__get_aliexpress_product",
      "mcp__fetchaller__search_aliexpress",
      "mcp__fetchaller__get_alibaba_product",
      "mcp__fetchaller__search_alibaba"
    ]
  }
}

Restart Claude Code.

Recommended CLAUDE.md Addition

Add this to your project's CLAUDE.md (or global ~/.claude/CLAUDE.md) to instruct Claude to prefer fetchaller:

## Web Fetching & Search

**ALWAYS use fetchaller tools instead of WebFetch and WebSearch.** fetchaller has no domain restrictions and produces cleaner output.

- `mcp__fetchaller__fetch(url, maxTokens?, timeout?)` — Fetch any URL → clean markdown
- `mcp__fetchaller__search(query, page?)` — Web search (Google + DuckDuckGo)
- `mcp__fetchaller__browse_reddit(subreddit, sort?, time?, limit?)` — Browse subreddit listings
- `mcp__fetchaller__search_reddit(query, subreddit?, sort?, time?, limit?)` — Search Reddit posts
- `mcp__fetchaller__search_marketplace(query, location, platforms?, category?, sort?, condition?, min_price?, max_price?)` — Search Kijiji + Craigslist + Facebook Marketplace
- `mcp__fetchaller__search_realtor(location, transaction?, property_type?, building_type?, min_price?, max_price?, min_beds?, min_baths?, ownership?, sort?, page?)` — Search realtor.ca homes
- `mcp__fetchaller__get_aliexpress_product(product_id)` — AliExpress product details
- `mcp__fetchaller__search_aliexpress(query, page?, sort?, min_price?, max_price?)` — Search AliExpress
- `mcp__fetchaller__get_alibaba_product(product_id)` — Alibaba.com product details
- `mcp__fetchaller__search_alibaba(query, page?, sort?, min_price?, max_price?)` — Search Alibaba.com

Usage

The mcp__fetchaller__fetch tool is now available:

# Fetch a URL
fetch https://example.com

# Fetch with token limit
fetch https://example.com maxTokens=10000

# Fetch slow site with longer timeout
fetch https://slow-site.com maxTokens=25000 timeout=60

Web Search

# Search the web
search "python asyncio tutorial"

# Page 2 of results
search "python asyncio tutorial" page=2

Web Research Pattern

1. Use search to find URLs
2. Use fetch to read them

Tool Reference

fetch(url, maxTokens?, timeout?, raw?)

Parameter	Type	Default	Description
url	string	required	URL to fetch (http/https)
maxTokens	number	25000	Max tokens to return
timeout	number	10	Request timeout in seconds
raw	boolean	false	Return raw HTML instead of markdown

Returns

Clean markdown with:

• Page title as H1
• Scripts, styles, nav, footer, iframes removed
• HTML converted to markdown
• Redirects noted
• Content truncated at token limit

Edge Cases

Scenario	Behavior
Invalid URL	Error message
Non-200 response	Error + partial body
JSON content	Returned as-is
XML/RSS feeds	Returned as-is
CSV files	Returned as-is
Plain text	Returned as-is
PDF files	Text extracted
Timeout	Error after timeout (default 10s)
Huge page	Truncated at maxTokens

search(query, page?)

Parameter	Type	Default	Description
query	string	required	Search query
page	number	1	Result page (1-indexed)

Searches Google (primary) and DuckDuckGo (supplement) in parallel. Returns titles, URLs, and snippets. Page 2+ queries Google only.

Reddit Tools

Two tools for Reddit research:

browse_reddit - Browse Subreddit Listings

browse_reddit({
  subreddit: "LocalLLaMA",   // without r/ prefix
  sort: "hot",               // hot, new, top, rising
  time: "day",               // hour, day, week, month, year, all (for "top" only)
  limit: 10                  // 1-25
})

Returns post titles, scores, comment counts, and URLs. Use fetch to read full posts.

search_reddit - Search Posts

search_reddit({
  query: "best mass spectrometry software",
  subreddit: "labrats",      // optional - limit to subreddit
  sort: "relevance",         // relevance, hot, top, new, comments
  time: "year",              // hour, day, week, month, year, all
  limit: 10                  // 1-25
})

Returns matching posts with metadata. Use fetch to read full discussions.

URL Transformation

All Reddit URLs are automatically transformed to old.reddit.com for 65-70% token savings. Trailing slashes are added to avoid 301 redirects (~50-100ms latency savings):

Input URL	Transformed To
www.reddit.com/r/foo	old.reddit.com/r/foo/
reddit.com/r/foo	old.reddit.com/r/foo/
old.reddit.com/r/foo	old.reddit.com/r/foo/

Rate Limits

Reddit allows ~10 unauthenticated API requests per minute. browse_reddit and search_reddit each use 1 API call. fetch uses HTML (no API call).

AliExpress & Alibaba Tools

get_aliexpress_product(product_id) - Product Details

Accepts a numeric product ID (e.g., 1005006027485365) or full URL. Returns price, specifications, ratings, and recent reviews via AliExpress's MTop API.

search_aliexpress(query, page?, sort?, min_price?, max_price?) - Search Products

Parameter	Type	Default	Description
query	string	required	Search query
page	number	1	Page number (1-indexed)
sort	string	"default"	default, orders, price_asc, price_desc
min_price	number	—	Minimum price filter
max_price	number	—	Maximum price filter

get_alibaba_product(product_id) - B2B Product Details

Accepts a numeric product ID or full URL. Returns tiered pricing, MOQ, lead times, supplier info, and specifications.

search_alibaba(query, page?, sort?, min_price?, max_price?) - Search B2B Products

Parameter	Type	Default	Description
query	string	required	Search query
page	number	1	Page number (1-indexed)
sort	string	"default"	default, price_asc, price_desc
min_price	number	—	Minimum price filter (USD)
max_price	number	—	Maximum price filter (USD)

Marketplace Search

search_marketplace — Search Kijiji, Craigslist, and Facebook Marketplace

Searches all three platforms concurrently with human-readable parameters and returns grouped results.

Parameter	Type	Default	Description
query	string	required	Search keywords
location	string	required	City name (e.g. "toronto", "st catharines, ON", "seattle")
platforms	string[]	all	Platforms to search: kijiji, craigslist, facebook
category	string	"all"	all, cars, electronics, furniture, clothing, tools, free, bikes, phones, motorcycles, boats, rvs, auto_parts, sporting, toys, baby
sort	string	"date"	date, price_asc, price_desc, relevance
condition	string	—	new, like_new, good, fair
min_price	number	—	Minimum price in dollars
max_price	number	—	Maximum price in dollars

Kijiji is Canada-only and automatically skipped for US locations. Location matching supports exact names, common aliases (e.g. "niagara" → Hamilton CL area), and fuzzy matching for typos.

Real Estate Search

search_realtor — Search realtor.ca homes

Searches Canadian homes for sale or rent via realtor.ca's api2, with the full filter set so an assistant can narrow a home search. Returns listings with price, address, beds/baths, size, agent, and a realtor.ca URL.

Parameter	Type	Default	Description
location	string	required	City, neighbourhood, or postal code (e.g. "Ottawa", "Orleans, Ottawa", "M5V")
transaction	string	"sale"	sale, rent
property_type	string	"any"	any, residential, condo, recreational, vacant-land, multi-family, agriculture, parking
building_type	string	—	house, duplex, triplex, townhouse, apartment, other
min_price	integer	—	Minimum price (sale) or monthly rent
max_price	integer	—	Maximum price (sale) or monthly rent
min_beds	integer	—	Minimum bedrooms
min_baths	integer	—	Minimum bathrooms
ownership	string	—	freehold, condo
sort	string	"newest"	newest, oldest, price-asc, price-desc
page	integer	1	Result page (~20 per page, up to 600 returnable)

Call fetch(url) on any listing URL for the full description, every property detail, and similar nearby homes. fetch also handles realtor.ca search/SEO/map pages (/{prov}/{city}/real-estate, /map) and all wellfound.com pages (startup job search, job detail, company profiles).

How It Works

1. Validates URL (http/https only)
2. Blocks private/internal IPs (SSRF protection with DNS rebinding prevention)
3. Fetches with browser-like TLS fingerprints via wafer (Rust/BoringSSL) — rotates Chrome versions automatically
4. If bot challenge detected: solves automatically (see Bot Challenge Bypass below)
5. Detects content type
6. For HTML: removes junk elements (nav, footer, ads, cookie banners), applies site-specific cleanup (25+ sites including GitHub, Reddit, HN, Wikipedia, Medium, Stack Overflow, Amazon, eBay, AliExpress, Alibaba, DigiKey, Mouser, realtor.ca, wellfound.com, plus Ashby/Greenhouse/Lever/Gem/Dayforce/Cornerstone/Workday/BambooHR/JazzHR/Work-at-a-Startup job boards with embed + white-label detection, and more), converts to markdown
7. For JSON/XML/CSV/text: returns raw
8. For PDF: extracts text
9. Truncates to token limit

Bot Challenge Bypass

fetchaller transparently bypasses bot challenges. First requests to protected sites take longer (total wall time = solve + fetch, typically 10-40s), but subsequent requests use cached cookies and are fast (~0.5s). The timeout parameter only controls the HTTP fetch — the browser solve has its own internal timeouts.

Supported Challenges

Challenge	Method	Speed
Alibaba Cloud WAF (ACW)	Inline Python solver	~1ms
Alibaba Cloud WAF (TMD)	Inline warming + browser	~5-10s
Cloudflare Managed Challenge	Patchright browser solver	~3-30s
Akamai Bot Manager	Patchright browser solver	~3-15s
Amazon rate-limit/CAPTCHA	Patchright browser solver	~3-10s
DataDome, PerimeterX, Imperva	Patchright browser solver	~3-10s
Kasada	Browser CT token + Python SHA-256 PoW	~3-10s
GeeTest v4 slide CAPTCHA	CV notch detection + drag replay	~5-15s
reCAPTCHA v2	Checkbox → audio (Whisper) → ONNX grid	~5-30s

All challenge solving is handled by wafer's BrowserSolver (Patchright-based). Cookies are cached per-domain so subsequent requests skip the challenge.

Requirements

Docker: Patchright's bundled Chromium is included in the image. The cookie-data volume persists solved cookies across restarts. No extra setup needed.

Local (stdio): Browser support (Patchright) is included by default. Run patchright install chromium after installing to download the browser binary. Docker includes it automatically.

Architecture

Content Processing

src/fetchaller/content/ handles HTML→markdown conversion. Each site module exports is_<site>(url), SELECTORS_LIST, and optionally strip_<site>_junk(soup) / postprocess_<site>(markdown):

• html.py — Generic pipeline + dispatch. Universal junk selectors, markdownify, whitespace cleanup. Generic JSON-LD Product fallback.
• amazon.py — All TLDs (.com, .ca, .co.uk, .de, etc.). CSS selectors, soup cleanup, regex post-processors.
• github.py — CSS selectors, URL transforms, file tree extraction, issue/PR/discussion extraction from embedded JSON.
• reddit.py — CSS selectors for old.reddit.com, URL transforms (www→old), post formatting.
• hackernews.py — CSS selectors, table unwrapping, story block reformatter.
• medium.py — CSS selectors (data-testid), HTML-based detection for unknown custom domains.
• huggingface.py — data-target attribute selectors, filter tag/button cleanup.
• stackoverflow.py — All Stack Exchange sites. CSS selectors, soup cleanup, regex post-processors.
• redflagdeals.py — RFD-specific CSS selectors, soup cleanup, regex post-processors.
• forums.py — Generic forum support (XenForo, vBulletin, phpBB, Discourse). RSS/Atom feed autodiscovery.
• wikipedia.py — CSS selectors for edit buttons, navboxes, TOC, reference lists.
• alibaba.py — Embedded JSON extraction (window.detailData, window.__page__data_sse10), soup cleanup.
• aliexpress.py — CSS selectors, soup cleanup, regex post-processors.
• craigslist.py — All city subdomains. CSS selectors, regex post-processors. Search URL detection for SAPI intercept.
• facebook_marketplace.py — URL detection only. GraphQL client in facebook_marketplace/ package.
• digikey.py — All TLDs. CSS selectors, soup cleanup. Behind Akamai (wafer handles). HTML fallback without API key.
• ebay.py — All TLDs. JSON-LD product extraction, search result DOM extraction (.s-item), regex post-processors.
• molex.py — JSON-LD Product extraction (additionalProperty specs). CSR site — specs only in structured data.
• mouser.py — All TLDs. CSS selectors, soup cleanup. Behind Akamai. HTML fallback without API key.
• soylent.py — Shopify store cleanup, inventory extraction from gsf_conversion_data.
• ti.py — Document viewer support for lazy-loaded datasheets.
• ashby.py / greenhouse.py / lever.py / gem.py / dayforce.py / cornerstone.py / workday.py / bamboohr.py / jazzhr.py / workatastartup.py — Job-board platforms. All preserve the source's own field names, enum values, and section titles. Each posting and (where supported) board listing is dispatched to the platform's API/JSON shell before the generic HTML pipeline. Five embed/white-label detectors run during the HTML phase so company career pages like synaptivemedical.com/job-openings (white-label Dayforce), skywatch.com/careers/ (Ashby <script src="…/embed">), avidbots.com/company/careers/ (BambooHR <div id="BambooHR">), and earthdaily.com/job-openings (JazzHR multi-tenant) are upgraded to structured ATS output instead of returning empty SPA shells. See docs/site-apis.md for endpoints and detection details.

Search

src/fetchaller/search/ — Google + DuckDuckGo combined. Result merging/dedup, 5-minute cache, CAPTCHA backoff. Uses wafer.AsyncSession(profile=Profile.OPERA_MINI).

Site-Specific API Intercepts

CSR sites where HTML scraping produces garbage are intercepted in fetch_url() and routed to structured APIs:

• Craigslist (src/fetchaller/craigslist/) — SAPI v8 client (sapi.craigslist.org). Up to 120 items/request with total count. Area IDs from page HTML, cached per hostname. Listing pages stay in HTML pipeline.
• Kijiji (src/fetchaller/kijiji/) — Unauthenticated Apollo GraphQL (kijiji.ca/anvil/api). Search + listing detail. Prices in cents.
• Facebook Marketplace (src/fetchaller/facebook_marketplace/) — GraphQL (facebook.com/api/graphql/). Geocoded search, listing detail with photos.
• AliExpress (src/fetchaller/aliexpress/) — MTop API for products (token bootstrap + MD5 signing). SSR HTML for search. Reviews from feedback.aliexpress.com.
• Alibaba.com (src/fetchaller/alibaba/) — SSR HTML with embedded JSON. No MTop API for international site.
• eBay — SSR search results extracted from .s-item DOM elements, formatted as numbered list.
• Mouser (src/fetchaller/mouser/) — Search API client. Requires MOUSER_API_KEY.
• DigiKey (src/fetchaller/digikey/) — OAuth2 client_credentials API. Requires DIGIKEY_CLIENT_ID + DIGIKEY_CLIENT_SECRET.
• Marketplace Search (src/fetchaller/marketplace/) — Unified orchestrator searching Kijiji, Craigslist, and Facebook Marketplace concurrently. Human-readable params mapped to platform-specific values. Auto-skips Kijiji for non-Canadian locations.
• Dayforce HCM (src/fetchaller/content/dayforce.py) — Posting detail from SSR'd __NEXT_DATA__. Board listing via CSRF-protected POST to /api/geo/{namespace}/jobposting/search (NextAuth /api/auth/csrf round-trip required). White-label deployments on company domains are detected via __NEXT_DATA__.runtimeConfig.BASE_URL and rewritten to the canonical jobs.dayforcehcm.com board URL.
• Cornerstone OnDemand (src/fetchaller/content/cornerstone.py) — SPA shell carries a JWT in csod.context. Posting from services/x/job-requisition/v2/requisitions/{reqid}/jobDetails; board listing POSTed to rec-job-search/external/jobs on the regional cloud host (us|eu|uk|au.api.csod.com).
• Workday (src/fetchaller/content/workday.py) — {tenant}.wd{1-103}.myworkdayjobs.com boards and postings. Posting GET /wday/cxs/{tenant}/{site}/job{externalPath}; board POST /wday/cxs/{tenant}/{site}/jobs paginated in batches of 20 (capped at 200).
• BambooHR (src/fetchaller/content/bamboohr.py) — {tenant}.bamboohr.com/careers. Board GET /careers/list; posting GET /careers/{id}/detail. Both return clean JSON unauthenticated. Widget embeds (<div id="BambooHR" data-domain="{tenant}.bamboohr.com">) on company sites are auto-detected and resolved to the tenant subdomain.
• JazzHR (src/fetchaller/content/jazzhr.py) — {tenant}.applytojob.com/apply. Board SSR'd HTML (.list-group .list-group-item); posting reads schema.org JobPosting JSON-LD. Company sites that reference one or more JazzHR tenants via JS (e.g. earthdaily.com/job-openings) are auto-aggregated into a combined board.

HTTP Transport (Wafer)

All HTTP is handled by wafer (~/code/wafer). Fetchaller does NOT contain bot protection, challenge solving, or TLS fingerprinting code. If a site blocks requests, fix it in wafer.

Remote Deployment (HTTP Mode)

Deploy fetchaller as a remote MCP server for Claude.ai, Claude Desktop, or any MCP client.

Quick Start

# Run with authentication
MCP_API_KEY=your-secret-key python -m fetchaller.main --http

# Or use Docker
docker compose up -d

Local Development

# Build and test locally
docker compose -f docker-compose.local.yml up --build

# Test endpoints
curl http://localhost:6000/health
curl -X POST http://localhost:6000/mcp \
  -H "Authorization: Bearer test-api-key-local" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'

Claude Code/Desktop Config

{
  "mcpServers": {
    "fetchaller": {
      "type": "streamable-http",
      "url": "https://mcp.fetchaller.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Claude.ai Custom Connector (OAuth)

For Claude.ai web/mobile with cross-platform sync:

1. Go to Settings → Connectors → Add Custom Connector
2. Name: fetchaller
3. URL: https://mcp.fetchaller.com/mcp
4. Leave Client ID/Secret blank
5. Enter your API key when prompted

Environment Variables

Variable	Default	Description
HTTP_PORT	6000	Server port (1-65535)
MCP_API_KEY	(required)	Bearer token for auth
MCP_SERVER_URL	http://localhost:$PORT	Public URL for OAuth
JWT_SECRET	(derived from API key)	Secret for OAuth tokens
RATE_LIMIT_REQUESTS	100	Requests/minute per IP
MOUSER_API_KEY	—	Mouser Search API key (free registration)
DIGIKEY_CLIENT_ID	—	DigiKey API client ID (free registration)
DIGIKEY_CLIENT_SECRET	—	DigiKey API client secret

Security

• SSRF Protection: Blocks localhost, private IPs, link-local addresses, and DNS rebinding services (nip.io, xip.io, etc.). Resolves hostnames to verify final IP addresses.
• OAuth 2.1: PKCE required for all token exchanges. Timing-safe comparisons for auth codes.
• Rate Limiting: Per-IP rate limiting with configurable limits.

Files

fetchaller-mcp/
├── pyproject.toml           # Python package config
├── src/fetchaller/          # Python source
│   ├── main.py              # Entry point
│   ├── server.py            # MCP server setup
│   ├── config.py            # Configuration
│   ├── http/                # HTTP server (FastAPI)
│   ├── tools/               # MCP tools (fetch, search, reddit, aliexpress, alibaba, marketplace)
│   ├── content/             # Content processing (HTML→markdown, site-specific cleanup)
│   ├── search/              # Web search (Google + DuckDuckGo)
│   ├── aliexpress/          # AliExpress MTop API client, product, search, reviews
│   ├── alibaba/             # Alibaba.com product and search extraction
│   ├── mouser/              # Mouser Search API client
│   ├── craigslist/          # Craigslist SAPI client + location resolution
│   ├── kijiji/              # Kijiji GraphQL API client + location resolution
│   ├── facebook_marketplace/# Facebook Marketplace GraphQL client
│   ├── marketplace/         # Unified marketplace search orchestrator
│   ├── digikey/             # DigiKey API client (OAuth2 + product/search)
│   ├── cache/               # Response caching
│   ├── queue/               # Reddit rate limiting
│   └── security/            # SSRF, crypto, XSS
├── docker-compose.yml       # Production deployment
├── docker-compose.local.yml # Local testing
├── Dockerfile               # Container build
├── docs/                    # Architecture & developer docs
├── CLAUDE.md                # Instructions for Claude
├── README.md                # This file
└── landing/                 # Static site (fetchaller.com)
    ├── index.html           # Landing page
    └── llms.txt             # LLM-readable project summary (llmstxt.org spec)

Dependencies

• wafer-py[browser] - HTTP transport with TLS fingerprinting, bot challenge bypass, and browser solver (Rust/BoringSSL + Patchright)
• mcp - MCP protocol SDK
• fastapi + uvicorn - HTTP server
• beautifulsoup4 + markdownify - HTML to markdown
• pymupdf4llm - PDF to markdown extraction
• pyjwt - OAuth tokens

Testing

# Run tests
uv sync --extra dev
.venv/bin/ruff check src/ tests/
.venv/bin/python -m pytest tests/ -x -q

# Test in Docker
docker compose -f docker-compose.local.yml up --build
curl http://localhost:6000/health

License

MIT