Selvage

The self-finished edge that keeps your scattered knowledge from unraveling.

A selvage is the woven edge of fabric that stops it from fraying. Selvage the project is the same idea for your team's knowledge: a thin, hybrid-searchable layer over Slack, wikis, code, and databases, queryable by humans, MCP clients, and agents. Self-hosted, local-first, MIT-licensed.

Why

Every growing team reinvents the same broken thing: a half-searchable wiki, a Slack history nobody can find, docs that quietly go stale. The usual fix, "put everything in one place," never survives contact with how people actually work. Data keeps living where it's produced.

Selvage takes the other path: a common interface over data wherever it already lives. Connectors normalize each source into one shared shape and land it in a single Postgres table. Retrieval knows nothing about where a record came from, so exact error codes and paraphrased questions both work, and the same query serves a person at a CLI, an agent over MCP, or the web UI.

• Pluggable connectors. A small contract, not a fixed integration list. Ships with filesystem/git-repo, Slack, and generic Postgres-table connectors. Writing your own is about 50 lines (guide).
• One unified store. Postgres + pgvector. Every source lands in the same documents table; nothing connector-specific leaks into retrieval.
• Hybrid retrieval. Full-text, vector, and rare-term lists, fused with Reciprocal Rank Fusion, recency-decayed, then cross-encoder reranked.
• Local by default. Embeddings and reranking run on your CPU via ONNX. The quickstart needs no API key, no cloud account, no tokens.
• MCP-native. Narrow, LLM-free tools any agent can compose, plus a minimal web UI that adds a plan → fan-out → synthesis loop for humans.

Quickstart

Five minutes, needs Node 20+ and Docker.

git clone https://github.com/naidx0/selvage.git && cd selvage
npm install
npm run setup          # starts Postgres+pgvector in Docker (port 5433), migrates schema
npm run sync           # indexes examples/docs via the filesystem connector
npx tsx cli.ts search "ERR_GRIPPER_TIMEOUT_0x2F"        # exact-token query
npx tsx cli.ts search "how do new engineers get set up" # semantic query
npx tsx cli.ts who-knows "gripper stalls"               # who has written about a topic

The first query in a process loads the local models (a few seconds, up to ~20s cold on a busy machine); the first run ever downloads ~40MB of ONNX models from the Hugging Face hub and caches them for later. No account required.

Point it at your own data

The quickstart indexes the sample corpus in examples/docs. To index your own, edit selvage.config.json and change the filesystem source's path to any folder of markdown, code, or text, then sync again.

Before (ships in the repo):

{
  "sources": [
    { "id": "example-docs", "connector": "filesystem", "config": { "path": "./examples/docs" } }
  ],
  "projects": {
    "handbook": { "description": "Example project scoping only the sample docs", "sources": ["example-docs"] }
  }
}

After (your handbook plus a code repo, bundled into one project):

{
  "sources": [
    { "id": "handbook", "connector": "filesystem", "config": { "path": "/Users/you/notes" } },
    { "id": "app-code", "connector": "filesystem", "config": { "path": "/Users/you/src/app" } }
  ],
  "projects": {
    "engineering": { "description": "Docs + code for the app team", "sources": ["handbook", "app-code"] }
  }
}

npm run sync                              # re-index everything
npx tsx cli.ts sources                    # list sources + document counts
npx tsx cli.ts search "your query" --project engineering

A project is a named bundle of sources that scopes a query so you don't get cross-domain noise. A source can live in many projects without being duplicated. SELVAGE_CONFIG points at a different config file if you don't want it in the repo root.

Connectors

A connector is one self-contained folder, connectors/<name>/index.ts, that implements a four-field contract: { name, syncMode, validateConfig, sync }. Core handles embedding, batching, upserts, change detection, pruning, and project scoping. Three ship in the box.

filesystem

Points at a directory and indexes every known-text file, chunked at language-aware boundaries (markdown at headings, code at declaration boundaries, prose at paragraphs). full sync, so deleted and renamed files drop out of the index on the next run. For markdown it also extracts an author (YAML frontmatter author: or a leading **Author:** line) so who-knows works on plain files.

{
  "id": "handbook",
  "connector": "filesystem",
  "config": { "path": "./docs", "exclude": [".git", "node_modules"], "maxFileSizeKb": 512 }
}

No secrets. exclude and maxFileSizeKb are optional (defaults shown).

slack

Polls conversations.history per channel and resumes from a durable cursor, so it runs fine from a cron job or a one-shot sync with just a bot token. It also walks thread replies. The bot must be a member of each channel.

{
  "id": "eng-slack",
  "connector": "slack",
  "config": { "channels": ["C0123ABC", "C0456DEF"], "lookbackDays": 180 }
}

Set the token in the environment (never in the config file):

export SLACK_BOT_TOKEN=xoxb-...

Required bot scopes: channels:history, channels:read, users:read, team:read. Override the env var name with tokenEnv if you prefer.

postgres

The "bring your own database" reference. Point it at a table (or arbitrary SELECT) and map columns onto the shared schema. Keyset pagination, full sync. examples/postgres-seed.sql is a ready-made sample table to try it.

{
  "id": "tickets",
  "connector": "postgres",
  "config": {
    "table": "support_tickets",
    "columns": { "id": "id", "content": "body", "title": "subject", "author": "agent", "timestamp": "updated_at" }
  }
}

Set the source database URL in the environment:

export SELVAGE_PG_SOURCE_URL=postgres://user:pass@host:5432/dbname

Override the env var name with connectionStringEnv. Provide exactly one of table or query; the id column must be orderable.

Write your own

The contract is the whole surface, and core never needs touching. docs/connector-authoring.md walks through a complete JSON-lines connector in about 50 lines, with the rules that keep a connector honest (secrets from env, fail loudly, pure exported normalization, log to stderr) and a pre-ship checklist.

Retrieval architecture

flowchart LR
  subgraph sources [Your data, where it already lives]
    FS[Folder / git repo]
    SL[Slack]
    PG[(Any Postgres table)]
    X[Your connector...]
  end

  subgraph selvage [Selvage]
    C1[Connectors: normalize]
    DB[(Postgres + pgvector<br/>unified documents table)]
    subgraph retrieval [Hybrid retrieval, LLM-free]
      FTS[full-text GIN]
      VEC[vector HNSW]
      RARE[rare-term IDF]
      RRF[RRF fusion k=60]
      DECAY[recency decay]
      RERANK[cross-encoder rerank]
      FTS --> RRF
      VEC --> RRF
      RARE --> RRF
      RRF --> DECAY --> RERANK
    end
    C1 -->|shared schema + embeddings| DB
    DB --> FTS & VEC & RARE
  end

  subgraph clients [Clients own orchestration]
    MCP[MCP server]
    WEB[Web UI]
    CLI[CLI]
  end

  FS & SL & PG & X --> C1
  RERANK --> MCP & WEB & CLI

Pure vector search loses on exact tokens like error codes and flag names, and short-but-rare messages sink under generic filler on cosine similarity. So Selvage runs three ranked lists per query, each over the resolved project or source scope:

1. Full-text over a stored tsvector with a GIN index, ranked by ts_rank_cd. Wins on exact tokens.
2. Vector, pgvector HNSW cosine, the query embedded with the BGE instruction prefix. Wins on paraphrase.
3. Rare-term, an IDF-style guard: query tokens that are rare in the corpus get their own list so a single exact match can't be drowned out.

The three lists fuse with Reciprocal Rank Fusion: score(d) = Σ 1/(60 + rank). RRF needs only ranks, which is exactly right when ts_rank_cd and cosine similarity live on incomparable scales. k=60 is the standard default from Cormack, Clarke & Büttcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods" (SIGIR 2009).

A recency decay factor in [0.5, 1] (half-life 90 days) discounts stale answers without erasing them, so an exact-token hit on an old postmortem still surfaces. Finally the fused top-12 goes through a local cross-encoder (Xenova/ms-marco-MiniLM-L-6-v2) that reads query and document together. Every result carries a signals object with the per-list ranks, RRF score, decay factor, and raw rerank logit, so any ranking is debuggable.

Embeddings are local ONNX (Xenova/bge-small-en-v1.5, 384-dim, q8) via @huggingface/transformers, on CPU, no API key. Full detail and the decision log: docs/architecture.md.

Demo

Real CLI output against the sample corpus (12 chunks), trimmed to the top hits.

Exact-token query lands on the error-registry chunk, with the matching postmortem right behind it:

$ npx tsx cli.ts search "ERR_GRIPPER_TIMEOUT_0x2F"

1. [0.997] example-docs · Fleet API reference › Error registry (excerpt)
   ## Error registry (excerpt) | Code | Meaning | | `ERR_GRIPPER_TIMEOUT_0x2F` |
   Servo watchdog fired during gripper actuation. | ...
   fts#2  vec#1  rare#2  rrf=0.0487  recency=1.00  rerank=8.11

2. [0.989] example-docs · Postmortem: fleet-wide gripper stalls, 2026-03-12
   # Postmortem: fleet-wide gripper stalls, 2026-03-12 **Severity:** SEV-1 ...
   fts#1  vec#2  rare#1  rrf=0.0489  recency=1.00  rerank=7.95

who-knows aggregates authored matches across sources:

$ npx tsx cli.ts who-knows "gripper stalls"

1. Priya Raman — 2 matches (sources: example-docs, latest 2026-07-17)

Add --json to any query for machine-readable output, --project <name> or --sources <a,b> to scope it, and --no-rerank to skip the cross-encoder.

MCP

Expose retrieval to Claude Code or any MCP client. The tools are deliberately narrow and LLM-free, so an agent can fan out over them cheaply and owns its own orchestration.

# Run from the repo root; the path must be absolute.
claude mcp add selvage -- npx tsx "$(pwd)/mcp-server/index.ts"

Tools exposed:

Tool	Does
selvage_search	Hybrid search, returns chunks with their citation uri
selvage_who_knows	Ranks people by how much they authored on a topic
selvage_list_sources	Lists connected sources with document counts
selvage_list_projects	Lists projects (search scopes) and their sources

selvage_search and selvage_who_knows take an optional project or sources scope. Smoke-test the server end-to-end anytime:

npx tsx scripts/mcp-smoke.mts

Web UI

npm run web            # http://localhost:3000 (override with SELVAGE_WEB_PORT)

Search mode is fully local, the same hybrid pipeline as the CLI. Ask mode adds a reference agent loop (plan → parallel tool fan-out → synthesized, cited answer) over the same tools; it needs ANTHROPIC_API_KEY set. Without the key, Search still works and Ask reports itself as disabled rather than failing. Model defaults to claude-sonnet-5 (SELVAGE_ANTHROPIC_MODEL).

The web UI is single-node and has no built-in auth. Run it on localhost or behind your own proxy.

Configuration

Everything has a working default except the secrets for optional connectors and Ask mode. Copy .env.example to .env or export in your shell.

Variable	Default	Purpose
DATABASE_URL	postgres://selvage:selvage@localhost:5433/selvage	Selvage's own Postgres (matches docker-compose.yml)
SELVAGE_CONFIG	selvage.config.json	Path to the source/project config
SELVAGE_EMBEDDINGS	local	local (ONNX, no account) or openai (any OpenAI-compatible /v1/embeddings)
SELVAGE_EMBED_MODEL	Xenova/bge-small-en-v1.5	Local embedding model
SELVAGE_EMBED_DIM	384	Embedding dimension
SELVAGE_EMBED_DTYPE	q8	q8 (fast) or fp32; part of the index identity
SELVAGE_HALF_LIFE_DAYS	90	Recency-decay half-life
SELVAGE_RERANK	true	Cross-encoder rerank of the fused top-N
SELVAGE_RERANK_MODEL	Xenova/ms-marco-MiniLM-L-6-v2	Cross-encoder model
SLACK_BOT_TOKEN	—	Bot token for a Slack source
SELVAGE_PG_SOURCE_URL	—	Source DB URL for a Postgres source
ANTHROPIC_API_KEY	—	Enables Ask mode in the web UI
SELVAGE_ANTHROPIC_MODEL	claude-sonnet-5	Model for Ask mode
SELVAGE_WEB_PORT	3000	Web UI port

OPENAI_API_KEY / OPENAI_BASE_URL apply only when SELVAGE_EMBEDDINGS=openai. Full annotated list in .env.example.

Repo structure

core/          shared schema, embeddings, chunking, retrieval (RRF, rerank, recency)
connectors/    one self-contained folder per connector (filesystem, slack, postgres)
mcp-server/    MCP tool exposure (stdio)
web/           minimal reference UI + agent loop
docs/          architecture + connector-authoring guide
examples/      sample corpus + postgres seed to try everything without real data
cli.ts         migrate / sync / search / who-knows / sources / projects

Limitations

Honest about what's proven and what isn't.

• Scale is unproven at the high end. Verified on a small corpus. CPU embeddings are fine for thousands of documents; hundreds of thousands is untested.
• English-only full-text. The Postgres FTS config is 'english'. Other languages still get vector and rare-term matching, but not the full-text list.
• Slack is unit-tested, not live-tested. The normalization and polling logic have tests, but the connector has not run against a real workspace. Polling also misses edits and deletions of already-indexed messages (fetches are "newer than cursor" only).
• access_scope is stored, not enforced. It's surfaced on every result but never used as a filter. Selvage is a retrieval layer, not an authorization system; callers that need enforcement read the label and drop what a user may not see.
• Single-node, no web UI auth. Run it on localhost or behind your own proxy.
• No migration versioning yet. The schema evolves by a fresh migrate.

Roadmap

• More connectors. Notion, GitHub, and an Obsidian-vault connector are the next targets. Want a different source? Open a connector request.
• Incremental-sync reconciliation so incremental connectors (Slack) can pick up edits and deletes, not just new records.
• Non-English full-text configs, selectable per source.

How this was built

Selvage was built and finished end-to-end with orchestrated AI agents: parallel exploration sweeps, phase-gated build waves with hard file-ownership boundaries so no two agents touched the same file, and three rounds of adversarial external review that converged to zero findings. A 37-test gate (9 of them DB-backed integration) stands over every change, and every claim in this README was executed live before publication. The full receipt, including the plan and what actually shipped, is in docs/finish-plan.md.

Contributing

Bug reports, connector requests, and PRs welcome. Start with CONTRIBUTING.md; adding a connector is the headline contribution path.

License

MIT. Fork it, extend it, ship it. See LICENSE.