MCP RAG Agent

A local-first AI backend that combines Model Context Protocol, Retrieval-Augmented Generation, and an agent runtime to answer questions from a private knowledge base.

The project runs an MCP server that exposes retrieval tools. A client-side agent connects to that server, lets the LLM decide when to call tools, retrieves relevant knowledge chunks, and returns grounded answers with source citations.

Features

• MCP server exposing retrieval tools
• CLI agent client
• Interactive chat mode
• RAG pipeline over a local knowledge base
• ChromaDB vector store
• Pluggable LLM providers (Ollama, AWS Bedrock)
• Pluggable embedding providers (Ollama, AWS Bedrock)
• Optional reranking pipeline
• Source-grounded answers with citations
• Modular service layout for retrieval, embeddings, chunking, vector stores, and rerankers

Example Usage

Ask a single question:

uv run python -m client.src.main "Explain how MCP tools work with RAG"

Start interactive mode:

uv run python -m client.src.main --interactive

Example interactive session:

MCP RAG Agent

Type a question and press Enter.
Type 'exit' or 'quit' to stop.

You: Explain how MCP tools work with RAG

Assistant:
MCP tools allow the agent to access external capabilities, such as document retrieval, through a standard protocol...

Sources:
- knowledge/mcp.md#3
- knowledge/rag.md#1

Architecture

flowchart TD
    User[User] --> Agent[LangGraph Agent]
    Agent --> Runtime[Agent Runtime]

    Runtime --> LLM["LLM Provider<br/>(Ollama / Bedrock)"]
    Runtime --> MCPClient[MCP Client]

    MCPClient --> MCPServer[MCP Server]

    MCPServer --> RetrievalTool["retrieve_documents()"]
    MCPServer --> OtherTools["Other MCP Tools"]

    RetrievalTool --> Embed["Embedding Provider<br/>(Ollama / Bedrock)"]
    Embed --> VectorStore[ChromaDB Vector Store]
    VectorStore --> Reranker[Reranker]
    Reranker --> Context[Context Builder]

    Context --> RetrievalTool
    RetrievalTool --> MCPServer
    MCPServer --> MCPClient
    Runtime --> Answer[Grounded Answer with Citations]

How It Works

1. The user asks a question through the CLI.

2. The client starts and connects to the MCP server.

3. The agent discovers available MCP tools.

4. The LLM decides whether it needs to call a retrieval tool.

5. The MCP server runs the RAG pipeline:

   • embeds the query
   • searches ChromaDB
   • optionally reranks retrieved chunks
   • builds a context payload
   • returns source metadata

6. The agent generates a final answer using the retrieved context.

7. The final response includes citations such as:

[knowledge/mcp.md#2]

Project Structure

client/
└── src/
    ├── main.py                 # CLI entry point
    ├── agent_runtime.py        # Agent loop and tool-calling runtime
    ├── mcp_client.py           # MCP client wrapper
    ├── tool_executor.py        # Executes MCP tools
    ├── tool_mapper.py          # Converts MCP tools to Ollama tool schema
    └── llm/                    # LLM provider implementations
        ├── base.py
        ├── ollama.py
        └── bedrock.py
        
server/
└── src/
    ├── server.py               # MCP server entry point
    └── tools/                  # MCP tool registrations

services/
├── rag/
├── chunking/                   # Text chunking strategies
├── dto/                        # API response DTOs
├── embedding/                  # Embedding providers
|      ├── base.py
|      ├── ollama_provider.py
|      └── bedrock_provider.py
├── loaders/                    # Document loaders
├── mappers/                    # Domain-to-response mappers
├── models/                     # RAG domain models
├── pipelines/                  # Indexing and retrieval pipelines
├── rerankers/                  # Reranking implementations
└── vectorstores/               # Vector store adapters

knowledge/                      # Local knowledge base documents
scripts/                        # Utility scripts
chroma_db/                      # Local ChromaDB data

Requirements

Required

• Python 3.13+
• uv

Optional

• Ollama (for local inference)
• AWS credentials with Bedrock access (for Bedrock providers)

Example:

ollama pull llama3.1

If your embedding provider uses a separate embedding model, pull that model too.

Setup

Install dependencies:

uv sync

Start Ollama:

ollama serve

Run a question:

uv run python -m client.src.main "Explain how MCP tools work with RAG"

Run interactive chat:

uv run python -m client.src.main --interactive

CLI Options

uv run python -m client.src.main --help

Available options:

question              Optional question to ask the agent
-i, --interactive     Start an interactive chat session
--model               Ollama chat model to use
--max-steps           Maximum number of tool-calling steps

Examples:

uv run python -m client.src.main "What is AWS Bedrock?" --model llama3.1

uv run python -m client.src.main --interactive --max-steps 8

Source Citations

The retrieval tool returns source metadata for every retrieved chunk:

{
  "context": "...",
  "sources": [
    {
      "document_id": "knowledge/mcp.md",
      "chunk_index": 2
    }
  ]
}

The agent is instructed to cite retrieved sources in the final answer using this format:

[document_id#chunk_index]

Example:

MCP lets an agent call external tools through a standard protocol, which makes the RAG system easier to separate from the LLM runtime [knowledge/mcp.md#2].

Sources:
- knowledge/mcp.md#2
- knowledge/rag.md#4

Why MCP + RAG?

Traditional RAG systems often tightly couple the agent, retriever, vector store, and application logic. MCP creates a cleaner boundary:

• The agent does not need to know how retrieval is implemented.
• Retrieval can be exposed as a reusable tool.
• Other MCP-compatible clients can use the same backend.
• The server can add more tools without rewriting the agent runtime.

Current Retrieval Tool

The MCP server exposes:

retrieve_documents(query: str, top_k: int = 20, top_n: int = 5)

It returns:

• retrieved context
• document IDs
• chunk indexes
• vector similarity scores
• rerank scores when available

Retrieval Evaluation

Run retrieval evaluation:

uv run python -m scripts.evaluate_retrieval

Run with custom retrieval settings:

uv run python -m scripts.evaluate_retrieval --top-k 8 --top-n 4

Write detailed results to JSON:

uv run python -m scripts.evaluate_retrieval --json-output scripts/eval/results/latest.json

Web API and Browser Demo

Install API dependencies:

 uv add fastapi uvicorn

Run the FastAPI server:

uv run uvicorn api.main:app --reload

Open the browser UI:

[http://localhost:8000](http://localhost:8000)

Open the interactive API docs:

[http://localhost:8000/docs](http://localhost:8000/docs)

API Endpoints

Health check:

curl [http://localhost:8000/health](http://localhost:8000/health)

Non-streaming chat:

curl -X POST [http://localhost:8000/chat](http://localhost:8000/chat)
-H "Content-Type: application/json"
-d '{"message": "Explain how MCP tools work with RAG"}'

Streaming chat:

curl -N -X POST [http://localhost:8000/chat/stream](http://localhost:8000/chat/stream)
-H "Content-Type: application/json"
-d '{"message": "Explain how MCP tools work with RAG"}'

Direct retrieval:

curl -X POST [http://localhost:8000/retrieve](http://localhost:8000/retrieve)
-H "Content-Type: application/json"
-d '{"query": "What is Model Context Protocol?", "top_k": 8, "top_n": 4}'

List available MCP tools:

curl [http://localhost:8000/tools](http://localhost:8000/tools)

The browser UI uses the streaming endpoint so the answer appears token-by-token while the final response is generated.

Configuration

Chat generation and embeddings are configured independently, allowing combinations such as Bedrock + Ollama or Ollama + Bedrock.

Create a local .env file from the example:

cp .env.example .env

Use Ollama for chat and embeddings

CLIENT_LLM_PROVIDER=ollama OLLAMA_CHAT_MODEL=llama3.1 RAG_EMBEDDING_PROVIDER=ollama OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Use Bedrock for chat and Ollama for embeddings

CLIENT_LLM_PROVIDER=bedrock BEDROCK_CHAT_MODEL=anthropic.claude-3-5-sonnet-20240620-v1:0 AWS_REGION=us-east-1
RAG_EMBEDDING_PROVIDER=ollama OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Use Bedrock for chat and embeddings

CLIENT_LLM_PROVIDER=bedrock BEDROCK_CHAT_MODEL=anthropic.claude-3-5-sonnet-20240620-v1:0
RAG_EMBEDDING_PROVIDER=bedrock BEDROCK_EMBEDDING_MODEL=amazon.titan-embed-text-v2:0
AWS_REGION=us-east-1

If you switch embedding providers, rebuild or re-index the vector database because vectors from different embedding models are not compatible.

Architecture Decisions

• MCP defines the boundary between the agent runtime and backend capabilities.
• LangGraph orchestrates workflows but does not implement retrieval logic.
• Retrieval is exposed as an MCP tool, allowing any MCP-compatible client to reuse the backend.
• LLM providers are swappable through a common interface (Ollama or AWS Bedrock).
• Embedding providers are independent of the chat model.
• Vector stores are abstracted behind a common interface (ChromaDB by default).
• Retrieval results are exposed as DTOs to avoid leaking internal domain models through the MCP API.

Portfolio Highlights

This project demonstrates:

• MCP tool design
• RAG architecture
• Provider abstraction (Ollama / Bedrock)
• vector search
• source-grounded generation
• agent tool-calling loops
• modular backend design
• Python async programming
• clean CLI UX

Future Improvements

Planned improvements:

• Docker Compose setup
• streaming responses
• structured logging
• query rewriting
• hybrid keyword/vector retrieval
• automated tests
• CI workflow