CCI Black Book MCP

An MCP server that does semantic search over one or more grow manuals (e.g. the CCI Black Book) — a corpus of scanned/image-heavy PDFs — and returns bounded, cited evidence packs for an MCP client (Claude Code, Codex, …) to answer from. Every hit is labeled with the book it came from, and you can scope a query to specific book(s).

Scanned books are mostly pictures. Naive PDF-RAG indexes only the OCR text layer and loses every chart, diagram, and photo. This server embeds both the text and a render of each page, so the visual content is first-class in retrieval.

Requirements

• A Voyage AI API key — the embeddings are the whole point, so this is required.
• One or more source PDFs (the CCI Black Book and/or other grow manuals).
• Docker, or Python 3.12 + uv.

How retrieval works

Three rankers, fused with weighted Reciprocal Rank Fusion:

• text-dense — voyage-context-4 contextualized chunk embeddings over the OCR text (one vector per chunk).
• image-dense — voyage-multimodal-3.5 over a PyMuPDF render of each page (one vector per page), so figures/photos/charts are retrievable even where the OCR layer is empty.
• FTS/BM25 — exact keyword matches over the OCR text.

Every page is rendered; a conservative, logged blank-page filter drops lined "Notes:" template pages (text-poor and ink-poor and color-poor) while keeping figure pages. Results carry unit_type = text or image plus a source_id/source_title; unit ids are namespaced per book (e.g. cci-black-book:p0042-img). Ingest fails loud if Voyage is unreachable (the prior index is left intact); queries degrade to FTS-only if a dense space is unavailable.

Quickstart

git clone https://github.com/dephekt/cci-blackbook-mcp
cd cci-blackbook-mcp

# 1. Secrets: a bearer token clients present, plus your Voyage key.
cp secrets/cci.env.example secrets/cci.env
#    then edit secrets/cci.env:
#      VOYAGE_API_KEY=...                     (from https://www.voyageai.com/)
#      CCI_VOYAGE_RETENTION_CONFIRMED=true    (after opting out — see Privacy)

# 2. Drop one or more PDFs into data/source/. Each file's real name becomes its
#    source id + title (e.g. "CCI Black Book.pdf" -> cci-black-book / "CCI Black Book").
mkdir -p data/source
cp "/path/to/CCI Black Book.pdf" "/path/to/Aroya Guide to Drying.pdf" data/source/

# 3. Build, start, and index (~$0.60 initially, ~8 min for a ~500-page book).
docker compose up -d --build
docker compose exec cci-blackbook cci-blackbook-ingest

docker compose up publishes the MCP on 127.0.0.1:8000 and serves the PDF/index from ./data — no reverse proxy or external network required. Point your client at http://127.0.0.1:8000/mcp (see Clients).

Verify Voyage connectivity first with cci-blackbook-ingest --smoke — it sends only synthetic data, so it's safe and essentially free.

Incremental ingestion and schema upgrades

Normal ingestion is source-incremental. It hashes each discovered PDF, compares its stored text and image pipeline fingerprints, and embeds only added or modified books. Removed books are deleted locally, and an unchanged corpus makes no Voyage calls. Every affected book rebuilds both embedding spaces in this first implementation.

Use make ingest for normal refreshes. make ingest-force is an explicit full rebuild that regenerates every embedding and may consume Voyage allowance or incur charges.

Schema-v3 indexes are never migrated or modified in place. Upgrade one offline:

docker compose stop cci-blackbook
docker compose run --rm cci-blackbook cci-blackbook-ingest --force
docker compose up -d cci-blackbook

The forced command builds and validates a separate schema-v4 database before atomically replacing the legacy file. Keep the MCP stopped for the complete one-off command.

Privacy

Ingest sends the book's OCR text and page images to Voyage's API. Voyage retains and may train on submitted data by default — turn on the one-way zero-retention opt-out in the Voyage dashboard, then set CCI_VOYAGE_RETENTION_CONFIRMED=true. Ingest is hard-gated on this flag and refuses to send anything otherwise.

Clients

Claude Code:

claude mcp add --transport http cci-blackbook http://127.0.0.1:8000/mcp \
  --header "Authorization: Bearer $CCI_BLACKBOOK_MCP_TOKEN"

Codex:

[mcp_servers.cci_blackbook]
url = "http://127.0.0.1:8000/mcp"
bearer_token_env_var = "CCI_BLACKBOOK_MCP_TOKEN"
tool_timeout_sec = 120

Tools

• ask_blackbook(question, crop_context=None, facility_context=None, max_citations=6, sources=None)
• blackbook_search(query, limit=10, mode="hybrid", sources=None) — modes: hybrid | vector | fts | text | image
• blackbook_read_citation(chunk_id) — accepts a namespaced text id (cci-black-book:p0042-c001) or page-image id (cci-black-book:p0042-img)
• blackbook_status() — lists every indexed book with per-book counts

sources scopes a query to one book id or a list of them (default: all); ids come from blackbook_status(). Each result carries source_id/source_title (the book) — distinct from the result's sources field, which lists the rankers (fts/text_dense/image_dense) that surfaced it.

Configuration

Env-driven; the compose file sets sensible defaults. The ones you'll actually set:

Variable	Notes
VOYAGE_API_KEY	required
CCI_VOYAGE_RETENTION_CONFIRMED	must be true to ingest (see Privacy)
CCI_BLACKBOOK_MCP_TOKEN	bearer token clients must send
CCI_SOURCE_DIR	directory of PDFs to index (default /data/source)
CCI_RENDER_DPI	100 — page render DPI for the image embeddings
CCI_RRF_WEIGHT_IMAGE	2.0 — upweights the image ranker in fusion

Deploy behind a reverse proxy

deploy/pangolin.yml is an optional overlay (Pangolin/Newt shown; adapt for any proxy). It drops the published host port and fronts the container's :8000:

DOMAIN=example.com CCI_DATA_DIR=/srv/cci-blackbook \
  docker compose -f docker-compose.yml -f deploy/pangolin.yml up -d --build

Development

make test   # offline unit tests — deterministic fixtures, no network or API key
make lint   # ruff