deduplicator-merge-advanced

A fair-merge engine for security-audit findings. When several auditors review the same codebase, they file overlapping findings for the same bug — described differently, from different angles, each with its own PoC or fix. This tool reads every finding, groups them by root cause, and produces a plan to merge them fairly: keep the most complete one, credit every contributor, and fold in what the others uniquely add.

It is not a duplicate finder. Finding duplicates is the easy half. The hard, valuable half is merging them without losing content or erasing authorship — that is what this does.

The problem: partial deduplication

Naïve deduplication treats findings as exact matches: find the "duplicates", close them, move on. Real audit findings are partially overlapping — same root cause, different manifestation, different code path, a unique PoC, a different fix angle. Closing them as duplicates:

• loses technical substance — the closed finding's extra affected function, its short PoC, or its alternative fix is gone; and
• erases authorship — the auditor who filed it silently loses credit.

Fair merging solves this. Every merge decision is designed so nothing of value is lost.

How the fair merge works

Each logical piece exists for a reason:

• Root-cause grouping, not title/label matching. The same bug is written five ways ("TOCTOU", "concurrent claim race", "double-tap"). Grouping reads the actual vulnerability, ignoring template Impact/Likelihood fields and [M-01]-style ID tags, which carry no signal.
• Relation types, because overlap is partial, not binary — exact_dup, partial_dup, subset, same_class, and overlapping_canonical. The last one protects a finding that is canonical elsewhere from being wrongly closed here.
• Canonical selection by completeness. The most complete finding becomes the base (scored on root-cause clarity, code location, attack scenario, PoC, exact-line fix).
• Author transfer — the fairness core. Every member's authors, including overlapping_canonical members, are credited on the canonical. No one's work vanishes.
• Content enhancement (edit TODOs). The canonical is augmented with each duplicate's unique substance — a missing affected function, a materially different fix, a short PoC — as surgical, copy-pasteable edit instructions.
• Severity recalculation. A weighted view across all members, not one submitter's arbitrary label.
• Cross-group protection. A broad finding that is canonical for its own root cause is never destroyed by being closed as a duplicate of another group.

Full write-up: methodology/find-duplicates.md.

Architecture

Findings in, a validated merge plan out; the plan renders to review artifacts or is executed on GitHub.

flowchart LR
  F["Findings<br/>(GitHub issues or files)"] --> IDX[index]
  IDX --> SUM["summarize<br/>normalized per finding"]
  SUM --> GRP["group<br/>by root cause"]
  GRP --> REF["refine<br/>canonical + edit TODOs"]
  REF --> ASM["assemble + validate"]
  ASM --> PLAN[("merge plan")]
  PLAN --> MD["merge_plan.md<br/>final_findings.md"]
  PLAN --> AP["--apply → GitHub<br/>(labels · close · comment · edit)"]

The grouping and refine stages are the model's judgment (the methodology); every other stage is deterministic Python — batching, validation, and the safety net that folds any dropped finding back into standalone.

Two ways to run it

Both share the same brain (the methodology); they differ in the frontend.

1. GitHub mode (MCP)

An MCP server exposing data-access tools so a Claude agent (in Claude Code / Desktop) runs the merge interactively against a GitHub audit repo where findings are Issues (severity label + author <handle> labels). The agent fetches the index, reads bodies on demand, and persists progress to a workspace so a session survives interruptions.

pip install -e ".[mcp]"
python -m dedup_merge.mcp.server        # stdio MCP server

2. Standalone mode (Docker)

A self-contained analyzer: point it at a GitHub repo or a local findings folder, and it writes the merge artifacts — no interactive agent, no write access to anyone's repo.

docker compose run --rm merge --repo owner/name --out /out
# or from local findings (a dir of *.md, or findings.json):
docker compose run --rm merge --input /findings --repo-name my-audit --out /out

Or without Docker:

pip install -e .
dedup-merge --input examples/findings --repo-name example-audit --out ./out

It writes three artifacts to the output directory:

File	What it is
merge_plan.json	The full machine-readable plan (groups, relations, canonical, scores, author transfers, edit TODOs, standalone, notes)
merge_plan.md	The reviewable action list — what to close, who's canonical, authors credited, and how to edit the texts
final_findings.md	The merged result — surviving canonicals with combined authors, plus standalone findings

Applying the merge (--apply)

By default the tool only plans. Add --apply (GitHub mode only) to execute the plan on the repo: transfer author <handle> labels onto each canonical, close the merged duplicates (never an overlapping_canonical), recompute the canonical's severity as a weighted vote, post a Deduplicator report comment, and fold the edit TODOs into the canonical's body. It needs a write-scoped GITHUB_TOKEN; without --apply the same invocation prints the exact actions it would take.

dedup-merge --repo owner/name --out ./out            # dry run — prints what it would do
dedup-merge --repo owner/name --out ./out --apply    # executes on GitHub

Authentication

One Anthropic credential is required — either works:

• ANTHROPIC_API_KEY — a standard API key, or
• CLAUDE_CODE_OAUTH_TOKEN — a Claude Code subscription token (sent as an OAuth bearer; the client adds the required beta header automatically).

GitHub mode additionally needs a GITHUB_TOKEN. See .env.example.

Model

Defaults to claude-sonnet-5 (cost-effective for batch analysis). Override with DEDUP_MERGE_MODEL — claude-opus-4-8 for maximum quality, claude-haiku-4-5 for the cheapest passes.

Example

Run the bundled synthetic set:

dedup-merge --input examples/findings --repo-name example-audit --out ./out

It groups the promo-race findings (#101 canonical, #102/#104 merged), keeps the unrelated validation bug (#103) standalone, credits all three authors on #101, and emits an edit TODO folding the webhook angle into the canonical's fix. Sample artifacts are in examples/output/.

Quality

Two layers, because the tool has a deterministic half and a model half:

• Tests (tests/) cover the deterministic plumbing — pipeline wiring, plan validation, source parsing — with a mock LLM, so they run in CI without any credential. Lint (ruff), types (mypy) and tests run on Python 3.10–3.12 in CI.
• Evals (evals/) grade the model's judgment against hand-labeled cases on four metrics: grouping F1, canonical accuracy, author-transfer recall, and relation-type accuracy, with per-case pass/fail thresholds. The bundled cases (a real merge, distinct-bug over-merge stress, and a no-duplicates set) currently pass at 1.0 across all metrics on claude-sonnet-5.

pytest                       # offline, no credential
python evals/run_evals.py    # LLM-graded, needs a credential

License

MIT.