⚡ Multi-Repository Code Search Engine

A production-grade code retrieval and search system engineered for querying and navigating multiple source code repositories simultaneously, designed and planned using the OpenSpec Spec-Driven Development framework. Ranked results (with repository, file, line, symbol, and graph metadata) are the integration boundary for external cloud LLM clients, which perform generation in their own environment.

---

🌟 Key Features

1. Multi-Repository Ingestion & Incremental Sync:

   • Manages local codebase directories and remote Git repositories.
   • Respects .gitignore rules and excludes binaries/lockfiles automatically.
   • SHA-256 hash tracking and Git commit detection for instantaneous incremental updates.

2. AST-Aware Semantic Code Chunking:

   • Language-aware structural parsing for Python, TypeScript/JavaScript, Go, Rust, Java, C/C++, HTML/CSS, SQL, and Markdown.
   • Preserves function, method, class, and interface boundaries.
   • Injects scope headers (// [Context] Repository | File | Scope | Imports | Doc).

3. Hybrid Dense + Lexical Indexing:

   • Dense Vector Search: Semantic subword feature vectors with cosine similarity + support for external embeddings (Gemini, OpenAI, Voyage AI, Ollama). Local Ollama embeddings default to qwen3-embedding:0.6b, loaded on demand and released when idle.
   • Sparse BM25 Search: Code-tailored tokenizer splitting camelCase and snake_case tokens with symbol boosting.
   • Reciprocal Rank Fusion (RRF): Merges dense and sparse rankings with exact identifier boosts.

4. Symbol Graph & Cross-Repository Dependency Linkage:

   • Extracts symbol definitions, callers, callees, and imports in SQLite.
   • Automatically maps frontend client API calls (e.g. apiClient.post('/api/v1/auth/login')) to backend API route handlers across different repositories.

5. Interfaces:

   • Modern Web UI: Hybrid Search as the primary query experience, repository manager, cross-repo API contract map, and code inspector drawer.
   • Model Context Protocol (MCP) Server: Exposes stdio tools (search_codebases, get_symbol_definition, get_call_hierarchy, list_repositories) to AI coding assistants (Antigravity, Cursor, Claude Code, Windsurf).
   • CLI: Fast terminal commands for indexing and searching.
   • REST API: POST /api/v1/search returns ranked code chunks for external cloud LLM consumers.

---

📂 OpenSpec Spec-Driven Planning

All specifications, architectural contracts, and task breakdowns are maintained under openspec/:

openspec/
├── config.json                     # OpenSpec project configuration
├── specs/                          # Living System Specifications (Source of Truth)
│   ├── repository-management.md    # Repo ingestion & git tracking
│   ├── ast-code-chunking.md        # AST semantic parsing & context injection
│   ├── hybrid-indexing.md          # Dense vector + BM25 lexical index
│   ├── symbol-graph-retrieval.md   # Call graph & cross-repo API linkage
│   ├── context-fusion-reranking.md # RRF fusion & citation packaging
│   ├── rag-generation.md           # LLM prompting & grounded citations
│   ├── mcp-server.md               # Model Context Protocol tools
│   └── api-and-web-ui.md           # REST & Web UI specifications
└── changes/
    └── 01-foundation-and-core-rag/ # Phase 1 Change Proposal
        ├── proposal.md             # Goals, scope, and motivation
        ├── design.md               # Technical architecture & contracts
        └── tasks.md                # Implementation checklist (Completed)

---

🚀 Quick Start

1. Register & Index Repositories

# Add a local repository
python3 main.py add auth-service ./fixtures/repo_auth_service

# Add another repository
python3 main.py add web-client ./fixtures/repo_web_client

# List all indexed repositories
python3 main.py list

2. Manage Repository Groups & Dependency Relations

# Create a repository group
python3 main.py group create platform --repos auth-service shared-schemas

# Declare a dependency edge: web-client depends on auth-service
python3 main.py relation add web-client auth-service

# Inspect relations for a repository
python3 main.py relation show web-client

# Search with group scoping and upstream dependency expansion
python3 main.py search "jwt token" --group platform --expand upstream --expand-depth 1

3. Search Across Repositories (CLI)

# Hybrid search across all codebases
python3 main.py search "login user authenticate"

# Search scoped to a group with upstream dependency expansion
python3 main.py search "How does authentication flow between web-client and auth-service?" --group platform --expand upstream

4. Launch the Interactive Web UI

python3 main.py serve --host 127.0.0.1 --port 8000

Open http://localhost:8000 in your browser.

5. Connect to AI IDEs via MCP (Model Context Protocol)

Add this MCP server entry to your AI IDE configuration (Antigravity / Cursor / Claude Code):

{
  "mcpServers": {
    "multi-repo-code-rag": {
      "command": "python3",
      "args": ["/Users/nick-work-pc/.gemini/antigravity/scratch/multi-repo-code-rag/main.py", "mcp"]
    }
  }
}

---

🧠 Embedding Model Runtime

The engine runs as a single instance per data directory and keeps the local embedding model resident only while it is working.

• Default model: qwen3-embedding:0.6b (install once with ollama pull qwen3-embedding:0.6b). Override with --embedding-model or $OLLAMA_EMBEDDING_MODEL.

• On-demand residency: the model is never loaded at startup. It loads on the first embedding of an indexing run or search, and is released once the last in-flight operation finishes and the idle grace elapses. Overlapping requests share one load and produce one release.

• Residency policy via --keep-alive or $EMBEDDING_KEEP_ALIVE:

  Value	Behavior
  (unset)	Release after 30s of inactivity (default)
  0	Release immediately after the last operation
  45s, 5m	Release after that idle grace
  always	Keep the model resident for the process lifetime

• Single instance: startup takes an exclusive lock on <data-dir>/.rag-instance.lock. A second instance fails fast with the owning pid; pass --allow-multi-instance to downgrade this to a warning.

• Inspect / release manually: GET /api/v1/models/status reports residency, active operations, policy, and index provenance. POST /api/v1/models/unload (or python3 main.py unload) releases the model, returning 409 busy while an operation is in flight.

Automatic reindex on model change

The dense index records the provider, model, and vector dimension that produced its vectors (<data-dir>/index_meta.json). When the configured embedding model changes — for example on upgrade from qwen3-embedding:4b (2560 dims) to the qwen3-embedding:0.6b default (1024 dims) — the affected repositories are automatically re-embedded before search results are served:

• chunk text, symbol graph, and BM25 lexical index are preserved (embedding-only pass, not a re-parse);
• progress is reported through the normal indexing progress output;
• provenance is written per repository, so an interrupted rebuild resumes with the repositories still outstanding;
• searches arriving during a rebuild get 503 reindexing instead of being scored against vectors from another model.

Rollback to the previous behavior: OLLAMA_EMBEDDING_MODEL=qwen3-embedding:4b EMBEDDING_KEEP_ALIVE=always restores the old model and always-resident policy; the provenance check then rebuilds back into the 4b vector space with no code change.

---

🏷️ Repository Groups & Dependency Relations Architecture

Topology & Domain Rules

• Named Repository Groups: Flat collections of repositories (e.g. core, platform, billing). Deleting a group never deletes underlying repositories.
• Directed Dependency DAG: Explicit dependency edges A -> depends on -> B. Adding an edge runs write-time cycle detection (raising DependencyCycleError on cycles).
• Scope Resolution: Combines explicit repository IDs and group members into a primary set, then expands along the graph in upstream (dependencies), downstream (dependents), or both directions up to expand_depth.
• Hop-Decay Ranking: Chunks retrieved from expanded repositories receive a score multiplier penalty (0.85 ** hops) to ensure primary repositories rank first.
• Provenance Metadata: Results originating from expanded repositories carry metadata (repo_relation='expanded', relation_direction, relation_hops) and are visually badged in the UI.

REST API Endpoints

Method	Endpoint	Description
GET	/api/v1/groups	List all repository groups and their members
POST	/api/v1/groups	Create a new repository group {"name": "...", "repo_ids": [...]}
DELETE	/api/v1/groups/{name}	Delete a repository group
POST	/api/v1/groups/{name}/members	Add members to group {"repo_ids": [...]}
DELETE	/api/v1/groups/{name}/members/{repo_id}	Remove a member from a group
GET	/api/v1/models/status	Embedding model residency, policy, and dense index provenance
POST	/api/v1/models/unload	Release models now (409 while an operation is in flight)
GET	/api/v1/repos/{repo_id}/relations	Get repository groups, direct dependencies, and direct dependents
POST	/api/v1/repos/{repo_id}/dependencies	Add dependency edge {"depends_on": "..."}
DELETE	/api/v1/repos/{repo_id}/dependencies/{target_id}	Remove a dependency edge
POST	/api/v1/search	Search with optional groups, expand, and expand_depth

MCP Tools

• manage_repository_relations: Actions create_group, delete_group, add_to_group, remove_from_group, add_dependency, remove_dependency.
• get_repository_relations: Returns relations for a single repository or the entire relation graph.
• search_codebases: Extended with optional groups, expand, and expand_depth arguments.

---

🧪 Running Tests

python3 -m unittest discover -s tests -p "test_*.py" -v

All unit and integration test suites pass verifying AST chunking, symbol extraction, cross-repo API detection, repository relation DAG & cycle detection, scope resolution & hop-decay retrieval, REST API handlers, MCP protocol, and end-to-end hybrid search retrieval.