Evident

An open-source, agent-agnostic extraction and fetch layer.

Turn any URL — or any question about the web — into verified, typed data with a transparent confidence score, callable by any AI agent through MCP, REST, or a native SDK.

Evident is not another scraper. Best-in-class open-source scraping/rendering engines already exist (Crawl4AI, Playwright). Evident orchestrates them behind a resilience ladder, scores every result's trustworthiness, and lets you extract structured data from any site — not just ones someone hand-wrote a parser for — via a versioned, community-contributable recipe system.

Full vision, architecture, and roadmap: docs/VISION.md.

Why

Most extraction tools give you clean text and let you figure out whether to trust it. Evident's whole design centers on one missing piece: every result carries a confidence score and a method explaining how it was produced, so an autonomous agent — not a human — can decide whether to act on it.

Quickstart

git clone https://github.com/evident-project/evident
cd evident
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

# Run the MCP server (stdio) — works with Claude Desktop, Claude Code, Cursor,
# or any other MCP-compatible client
evident-mcp

Add to your MCP client config (example for Claude Desktop):

{
  "mcpServers": {
    "evident": {
      "command": "/absolute/path/to/.venv/bin/evident-mcp"
    }
  }
}

Not using an MCP client? Same engine, plain Python:

import asyncio
from evident.core import ladder

async def main():
    result = await ladder.run("https://example.com")
    record = ladder.to_record(result)
    print(record.confidence, record.method)
    print(record.data.get("markdown", "")[:500])

asyncio.run(main())

Tools (MCP) / functions (SDK)

Tool	What it does
fetch(url, mode)	Universal fetch, escalates the resilience ladder automatically
extract(url, json_schema)	Structured extraction against any caller-supplied schema — works on any site
list_recipes()	Discover built-in, verified extraction recipes
use_recipe(recipe_id, slug, entity_name)	Invoke a deterministic, high-confidence recipe (e.g. ats_greenhouse)
health_check(target)	Proactively check whether a recipe or URL is still working

Optional dependencies

Evident's core (Tier 1 static fetch) has minimal dependencies on purpose. Heavier capabilities are opt-in:

pip install "evident[render]"   # Tier 2: JS-rendered pages via Crawl4AI/Playwright
pip install "evident[llm]"      # extract(): LLM-based schema extraction (bring your own ANTHROPIC_API_KEY)
pip install "evident[api]"      # REST API interface
pip install "evident[all]"      # everything, plus dev/test tooling

If render isn't installed and Tier 1 fails, fetch() reports failure_reason: dependency_missing instead of crashing — Tier-1-only installs stay fully usable for the large share of the web that's server-rendered.

Contributing a recipe

The easy on-ramp for contributors is a recipe, not the core engine. A recipe is one YAML metadata file plus one small async fetcher function — see docs/RECIPE_GUIDE.md.

Testing

pip install -e ".[dev]"
pytest

Tests use recorded/mocked HTTP responses (respx) so they run deterministically without live network access — this was the single biggest gap in earlier hand-rolled scraping projects this one grew out of, and it's non-negotiable here.

Status

Early / pre-1.0. Tier 1 (static fetch) and the recipe registry (Greenhouse, Lever, Ashby) are implemented and unit-tested against mocked fixtures. Tier 2 (rendered fetch via Crawl4AI) is implemented and has been smoke-tested against a live page. LLM-based extract() is implemented but requires your own ANTHROPIC_API_KEY and hasn't been live-tested end-to-end yet — see docs/VISION.md roadmap for what's next.

License

Apache-2.0 — see LICENSE. Deliberately not AGPL, to stay commercial-use-friendly.