provgraf

A "bank of verified facts" for AI agents — a W3C PROV knowledge graph on plain Postgres, with automatic staleness propagation.

⚠️ This is a proof-of-concept / engineering demo, not a supported product. It was designed, evaluated and deployed for one real client engagement (anonymised here), and is published to document the idea and its implementation. Expect rough edges; issues are welcome, a roadmap is not promised.

The problem

If you let an AI agent write client-facing documents, the model is not the risk — the facts are. A language model will happily "correct" a true, contract-grade number (a rent rate, a buyout rule, a share-capital figure) toward whatever the internet believes. In our real deployment, an overnight agent once tried to fix six verified facts to match plausible-but-wrong public sources.

Not every certain fact arrives as a PDF. Plenty of what you are told is true and consequential — a director confirms on a call how a scheme works, and you act on it. People also change their minds. So a source here is either a document with a file, or a testimony: a dated, attributed record that on this day this person vouched for this. That is what lets you stop relitigating a settled question, and it is why the integrity check differs by shape rather than demanding a file for everything.

There is a second, more mundane version of the same problem. An agent left to its own devices dumps everything into Markdown files as it goes — notes, half-verified numbers, copies of copies — with no validation at the door. If you're disciplined about your files, maybe you can live with that. If you're messy (I am), the workspace silently rots: three files disagree about the same number and none of them says where it came from. That was the actual trigger for building this.

provgraf is the counter-measure: a small database of atomic facts, each with provenance (which official document it came from, who put it there, when), plus one mechanism you don't get from a notes file:

Change a source → the system tells you everything downstream that just went stale.

It cuts both ways: the validating write path keeps the human's workspace from rotting, and the read-only agent surface keeps the agent honest — it can quote the bank, but it can't quietly "improve" it.

The idea in 60 seconds

• Facts are nodes (entity), sources are nodes, people/software are agents, decisions are activitys — W3C PROV core plus documented extensions (provenance_class, entity_status; PROV sanctions extension via prov:type subtyping).
• Derived facts (sums, report figures) link to their inputs via wasDerivedFrom and store an inputs_hash of the input values.
• When a source fact is revised, a recursive SQL CTE walks the derivation graph and flags every transitively dependent fact as stale (measured: a 2-level cascade in ~2.5 ms).
• Versioning is a single trick: valid_to + a partial unique index (WHERE valid_to IS NULL). That gives as-of queries over the bank's own history for free.
• The bank is bitemporal: valid_from/valid_to is transaction time (what the bank believed when), world_valid_from/world_valid_to is world time (when the fact holds per its source). Only both axes together answer "per what we knew on 15 June, what was in force in May" — the question a backdated correction creates. The four-timestamp model is borrowed from Graphiti.
• Contradictory sources coexist as disputed alternates; a human resolves them with a recorded decision (the rejected alternative stays in the graph as a trail).
• No triplestore, no graph database. Postgres gives constraints, transactions, pg_dump, and a recursive CTE is all the graph traversal this problem needs.

graph LR
    D1["📄 datasheet<br/>(source doc)"] --> F1["riverside.units = 152"]
    D1 --> F2["hillside.units = 84"]
    D1 --> F3["lakeside.units = 58"]
    F1 --> T["units_total = 294<br/>inputs_hash ✓"]
    F2 --> T
    F3 --> T
    T --> R["report.units = 294"]
    D2["📄 NEW datasheet<br/>units 152 → 154"] -.revise.-> F1
    F1 -.cascade.-> T
    T -.cascade.-> R
    style D2 fill:#fdd,stroke:#c33
    style T stroke:#c33,stroke-dasharray: 5 5
    style R stroke:#c33,stroke-dasharray: 5 5

What's implemented

Layer	What it does
Core graph	CLI (typer + asyncpg): add, derive, revise, link, check, conflicts, resolve, subgraph, diagram (Mermaid), snapshot
Staleness engine	inputs_hash + recursive CTE cascade; the hash is computed identically in Python and SQL (hashing.py ↔ 03_staleness_fns.sql, parity is unit-tested, down to COLLATE "C" sort order)
Integrity invariants	No fact without provenance, no cross-client derivation, DAG guard (cycle rejection), duplicate detection — enforced in the database, they actually block
Bitemporal versioning	transaction time (valid_from/valid_to) + world time (world_valid_*); get <qname> --at … --world-at …, --history
Conflicts & decisions	disputed alternates → human resolve with recorded basis; recency-based suggestions are a hint, never auto-applied
Binding layer	prov:Collection nodes + hadMember from a per-client config/structure.json; open structural questions as first-class nodes with a resolution path
Semantic search (RAG)	Local sentence-transformers embeddings + a cross-encoder reranker; retrieval glosses are auto-generated from a field dictionary (config/gloss.json) — contextual retrieval without hand-writing descriptions. Provenance is deliberately excluded from the embedded text (it blurred the vectors).
MCP server	Read-only tools for AI agents (list_facts, get_fact, search, precedents, check) over stdio or SSE; lazy model loading + idle unload. Writes stay CLI-only — the architecture, not a prompt, enforces "no fact enters the bank without a human OK".
Agent write gating	Even on the CLI, a revision made by an agent of kind='software' lands as to_confirm until a human runs verify. An agent may propose; it cannot silently change a verified number.
Shared-source guard	check separates a missing source document from ORPHANED facts — those whose only source is that document. Facts backed by another live source are not flagged.
Two shapes of a certain source	A file-backed document (resolution, permit, registry extract) is verified by the file still being there. A testimony — someone competent vouched for it on a call — has no file by design; what makes it a record is who and when, and check flags a testimony missing either.
Interop	PROV-JSON export round-trips through the reference W3C prov library — covered by tests/test_prov_export.py, not just claimed
Dashboard	Streamlit view: facts, graph, documents, gaps
One report, two renderers	report.gather() computes check once; the CLI paints it and the MCP server serialises it. The human and the agent cannot end up looking at different states of the same bank.

How it was evaluated (the part that mattered)

The build was gated, not vibes-driven:

1. PRD first, reviewed by a panel of independent AI reviewer agents; their objections (e.g. "at ~30 facts Postgres barely beats a JSON file") were recorded as open tensions, not deleted.
2. A 1-day spike on flat files to prove the staleness-cascade design before writing any DDL.
3. A go/no-go milestone on real data: cascade correctness, version windows, invariants that actually reject bad writes, hash parity Python↔SQL.
4. Standards check: the PROV-JSON export deserializes cleanly with the reference W3C library, enforced by a test. Scope stated honestly: that proves the serialization is well-formed PROV-JSON, not PROV-CONSTRAINTS conformance (typing, causality loops, provenance travelling backwards in time), which needs a validator we have not run.
5. In production the bank grew to ~150 facts from 22 source documents and was used to fill investor-facing and grant documents, with a validator that blocks any hard number lacking a bank qname tag.

Quickstart (demo)

Requirements: Docker, uv.

docker compose -f infra/postgres/docker-compose.yml up -d --wait
uv run provgraf init
uv run pytest                      # 27 tests: hash parity, invariants, cascade, as-of, conflicts
bash examples/demo_cascade.sh      # seed a fictional company → check → revise a source → watch the cascade

(Run the tests before the demo — they read global check state, so a seeded database makes two of them fail. examples/reset.sh wipes and re-seeds; it connects through DATABASE_URL and refuses to touch a database holding anything other than the demo owner.)

The demo seeds a fictional social-housing company ("Acme Community Housing"), builds a 2-level derivation chain, plants a source conflict and an overdue fact, then revises one source number and shows check flagging the transitively dependent facts. It closes on the bitemporal query: a rent recorded today but in force since 1 June answers for June and correctly finds nothing in force in May.

Optional extras:

uv sync --group rag                # local embeddings + reranker (configure models in .env)
uv run provgraf embed acme-housing && uv run provgraf search "how much is the rent"
uv run --group dashboard streamlit run dashboard/app.py
uv run --group mcp provgraf-mcp    # read-only MCP server for AI agents

The default embedding/reranker models in .env.example are Polish (sdadas/mmlw-retrieval-roberta-large, sdadas/polish-reranker-large-ranknet) because the original deployment was Polish-language; swap them for any sentence-transformers-compatible pair.

Design decisions worth stealing

• Boring storage was the right call — and historically the norm. PROV is a data model, not a technology choice, and most deployed provenance recorders (Karma, Komadu, the IVOA provenance store) were relational too. Postgres gives constraints, transactions and pg_dump; a recursive CTE is all the graph traversal this problem needs. Presenting "PROV without a graph database" as a discovery would be a tell that you hadn't read the field.
• Hash parity enforced by tests. The staleness hash exists in Python and in a SQL function; a unit test feeds both the same fixtures. Divergence = the whole staleness feature silently lies.
• Read/write asymmetry for agents. Agents get a read-only MCP surface; writes go through a validating CLI with a human in the loop. Prompts can't enforce this — architecture can.
• Auto-glosses for retrieval. One dictionary entry per field type generates the embedded description for every fact of that type. New field → one JSON entry, not N hand-written descriptions.
• Decisions are nodes. Resolving a conflict creates an activity with an agent and a basis; precedents searches past decisions before you resolve a new dilemma.

Prior art, and what is actually different

Before publishing this I ran a prior-art survey across ~20 systems. Every ingredient here exists somewhere; the assembly is what does not. Being specific about that is more useful than a novelty claim:

• Graphiti / Zep — the closest neighbour and the one that beat us on an axis: genuine bitemporality with four timestamps per edge (which is why provgraf now has it). It has per-fact provenance back to the ingested episode. What it does not have is a staleness cascade — and its conflict handling is the mirror image of this project's: a small LLM decides at write time which contradicting fact loses and silently expires it, and its MCP surface hands the agent add_memory, delete_entity_edge and clear_graph.
• Dagster — hashes code and input data versions to mark downstream assets stale. That is inputs_hash + cascade, shipping and battle-tested, at the granularity of assets and tables. provgraf's difference is the unit (one number, with a citation) and the human in the loop, not the mechanism.
• Truth maintenance systems (Doyle 1979, de Kleer 1986) and content-addressed build systems (Nix, Bazel) are the real ancestors of justification-plus-invalidation. Nothing here is new under the sun; it is applied to facts instead of beliefs or build artifacts.
• TrustGraph — real prov: vocabulary in shipped code and a provenance CLI. Instructive twice over: it wrote this project's staleness feature down as a motivating use case and never built it, and it deliberately retreated from per-triple provenance to per-chunk containment because reification got expensive.
• WhyHow.AI — shipped per-triple → chunk → page-offset provenance back in 2024. The repositories have been untouched since, and the domain no longer resolves. Building this is demonstrably possible and demonstrably not sufficient on its own.
• Knowledge-base tools (Notion, Guru, Slab, Slite, Document360, GitBook) — verification exists, but the trigger is a calendar interval or an LLM judging that a document drifted from a connected source, and the unit is a page or a card. None of them holds a typed number, and none computes transitive dependents of a changed input. The industry is converging on probabilistic drift detection with accept/dismiss; the cascade here is deterministic and replayable. (Guru's "an edit by a non-owner unverifies the card" is where this project's agent gating comes from.)
• Agent memory (Mem0, Letta, LlamaIndex, LangMem, Google Memory Bank) — contradiction is resolved by overwriting or deleting, decided by a model. Mem0's prompt says it outright: if the retrieved facts contradict the memory, delete it. Coexisting disputed alternates plus a recorded human decision with a basis appears nowhere in that category.
• Not database provenance in the Green–Karvounarakis–Tannen sense (ProvSQL, GProM). That field annotates query results with semiring lineage; this is retrospective/workflow provenance. Different camp, easy to confuse.

Uncontested, as far as the survey found: fact-level staleness propagation, and conflicts that survive as alternates until a human records a decision. Everything else in the list above is prior art we are standing on.

Known limitations

Honest list (an adversarial code review ran before publishing; the notable leftovers):

• Clean PROV-JSON deserialization is tested; PROV-CONSTRAINTS conformance is not checked.
• No incremental recomputation: the cascade tells you what went stale, it does not recompute derived values for you.
• Single-user by design: no auth, no concurrency story beyond Postgres transactions.
• No rm-doc: documents are never deleted, only reported as dangling (with the orphaned facts they would take down).

Not built (on purpose)

• Query-time provenance — recording which facts fed a particular answer or document (TrustGraph does this). Likely belongs in the application consuming the bank rather than in the engine.
• LLM-resolved contradictions and auto-repairing memory. Both are well-trodden elsewhere and both defeat the point of a bank whose contents a human vouched for.

License

MIT