codesearch

Semantic search over local source repositories and forum/mailing-list archives, exposed both as a shell CLI and as an MCP server so any LLM (Claude Code, a local model, whatever speaks MCP) can look things up while helping you debug.

This is the retrieval half of warpembot, extracted and made project-agnostic. No email, no GitHub, no auto-replying — just indexing and querying.

The csbot command

One entry point for everything:

./csbot init warp https://github.com/warpem/warp.git --kb archives/warp-group.mbox

Registers a codebase (cloning it if you give a git URL, or just pointing at a local checkout), ingests any accumulated discussion archive you have for it, and builds the embeddings. --kb is optional — with no archive you get a code-only knowledge base. Missing archives are reported and skipped, not fatal.

./csbot update --pull

git pulls each registered repo and re-embeds only the files whose checksums changed. Drop --pull to index local edits; name repos to do a subset.

./csbot ask "why does frame series alignment deselect items" --repo warp

Retrieves the relevant source and prior discussions, sends them to a local Ollama chat model, and prints the answer followed by the file:line citations it drew on. Useful flags: --no-forum (code only), -k N (more context), --model (override the chat model), and --context-only, which prints the assembled context instead of generating — paste it into Claude or any other model when you want a stronger answer than the local one.

./csbot status
./csbot kb warp-group archives/more-threads.mbox

status shows chunk/document counts per source and flags repos whose path has gone missing. kb adds discussion knowledge to an existing project later.

Two models are involved and they are configured separately in config.json: embedding_model (used for indexing and search — changing it invalidates every index) and chat_model (used only by csbot ask to write the answer).

Components

The csbot wrapper is thin; the pieces underneath stay usable on their own.

File	Purpose
csbot	Single CLI entry point: init, update, ask, kb, status
cs_common.py	Config, Ollama embeddings, on-disk vector index, chunk metadata
cs_index.py	Build/update indexes: repos (incremental), mbox archives, text/JSON dumps
cs_server.py	MCP stdio server: list_sources, search_code, read_code, search_forum
cs_query.py	Shell CLI: code, forum, context
config.json	Which repos and corpora exist, embedding model, chunking params

Everything is local: embeddings come from Ollama, indexes are .npz files of L2-normalized vectors, search is a dot product. No external services.

Setup

pip install -r requirements.txt

ollama pull embeddinggemma:300m

embeddinggemma:300m is the default embedding model: ~600 MB and 768-dim, which keeps indexing and query latency sane on Apple silicon.

Vectors from different embedding models are not comparable, so changing embedding_model invalidates every index. index/manifest.json records which model built the indexes: csbot update refuses to write into a mismatched index, and the query paths warn instead of returning nonsense. To switch models, delete index/ and rebuild.

Each family also wants its own query/document prefixes (Qwen's Instruct: block, Gemma's task: search result | query:, Nomic's search_query:). These live in EMBED_PROFILES in cs_common.py and are selected automatically from the model name — add an entry there if you use something else.

Because embeddinggemma's context window is 2048 tokens, chunk_size defaults to 100 lines with 25 lines of overlap. If you move to a long-context embedder such as nomic-embed-text (8192), raising chunk_size back to 200 is safe.

Then use csbot init to register codebases, or edit config.json by hand:

• repos — name → path of each checkout you want searchable (relative paths resolve against this directory; ~ works).
• corpora — name → directory holding forum/mailing-list documents as JSON.

Direct access to the layers

csbot covers the normal workflow; these are the same operations unwrapped, for scripting or partial re-indexing.

Indexing

python3 cs_index.py repos

Walks every configured repo, chunks source files into overlapping line windows (chunk_size/chunk_overlap), and embeds them. It checksums each file, so re-running only re-embeds what changed — run it after a git pull.

python3 cs_index.py mbox warp-group archive.mbox

Imports a mailing-list archive (Google Groups exports an mbox) into a corpus. cs_index.py text <corpus> <dir> does the same for a directory of .txt, .md, or .json posts — use it for scraped forum threads; JSON items may carry id, title, author, date, body, url.

python3 cs_index.py status

Shows chunk/document counts per source and flags repos whose path is missing.

Querying from the shell

python3 cs_query.py code "how are CTF parameters fitted" --repo warp --show

python3 cs_query.py forum "tilt series stuck at 0%"

python3 cs_query.py context "why does frame series alignment deselect items" --repo warp --forum

context prints a paste-ready block — question, matching source with line numbers, optionally prior discussions — for feeding to a model that has no tools of its own.

Querying from an LLM (MCP)

.mcp.json registers the server — edit the placeholder path to point at your clone, then copy the block into any project's .mcp.json (or your MCP client's config) and the model gets four tools:

• list_sources() — what's indexed
• search_code(query, repo="", top_k=5) — file + line range + score
• read_code(repo, file, start_line, end_line) — the actual lines
• search_forum(query, corpus="", top_k=5) — prior discussions

The server loads indexes into memory at startup, so restart it after re-indexing.

Guidance worth putting in your CLAUDE.md (or system prompt): search code with a targeted query derived from the question, read the real source before answering, and treat forum hits as leads rather than truth — archived answers go stale when the software changes.

Reusing the warpembot data

The 2188-message Google Group corpus is already in corpora/warp-group/, copied from ../warpembot-public/threads/. load_doc maps the old message_id/subject/sender keys onto id/title/author, so it needs no conversion — only re-embedding:

python3 cs_index.py rebuild warp-group

The prebuilt .npz indexes that shipped with warpembot were built with qwen3-embedding:8b and are therefore unusable under the current model. They are parked in index_qwen8b_legacy/ — delete that directory once the rebuild is done, or restore it if you ever switch back.