MCP Chat

MCP Chat is a command-line interface application that enables interactive chat capabilities with AI models. It can run against either the hosted Anthropic API or a fully local model (Ollama via a LiteLLM proxy). The application supports document retrieval, command-based prompts, and extensible tool integrations via the MCP (Model Control Protocol) architecture.

Prerequisites

• Python 3.9+
• Either an Anthropic API key (hosted) or Ollama installed (local, no key required)

Running with a local model (no API key)

The app talks to the Anthropic SDK; a small LiteLLM proxy translates that to a local model, so no code changes are needed to switch backends — only .env. Two local backends are supported (defined in litellm_config.yaml): llama.cpp (default) and Ollama. Both serve the same qwen2.5:14b weights.

Common to both:

• .env is already set for local use:

  ANTHROPIC_BASE_URL="http://localhost:4000"
  ANTHROPIC_API_KEY="not-needed"
  

• Start the LiteLLM proxy in its own terminal and leave it running:

  uv run litellm --config litellm_config.yaml
  

• Run the app in another terminal:

  uv run main.py
  

Memory note: a 14B model at Q4 (~9 GB) runs comfortably on 24 GB+ of RAM. On 16–18 GB it works but is memory-tight — only one 14B can be GPU-resident at a time, so don't run llama.cpp and Ollama simultaneously. For lighter machines use a 7B variant.

Backend A — llama.cpp (default: CLAUDE_MODEL="qwen2.5-llamacpp")

1. Install: brew install llama.cpp
2. Get the weights. If you've already pulled the model with Ollama (below), llama.cpp can load that exact GGUF blob — no second download. Otherwise download a qwen2.5-14b-instruct GGUF from Hugging Face.
3. Start llama-server (leave running). Flash-attention + quantized KV cache are required to fit a 14B on ~18 GB:

   llama-server \
     -m ~/.ollama/models/blobs/sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 \
     --jinja -c 4096 -np 1 -fa on \
     --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 -ngl 999 \
     --host 127.0.0.1 --port 8080 --alias qwen2.5-14b
   

   --jinja enables tool calling (the app relies on it). The blob path is the GGUF Ollama stored; check yours with ls -lhS ~/.ollama/models/blobs/.

Backend B — Ollama (set CLAUDE_MODEL="qwen2.5-local")

1. Install and start: brew install ollama && brew services start ollama
2. Download the model: ollama pull qwen2.5:14b

Ollama runs as a background service (no separate terminal needed) and handles flash-attention/KV settings itself.

Running with the hosted Anthropic API

In .env, comment out ANTHROPIC_BASE_URL, set a real key, and use a real model id:

CLAUDE_MODEL="claude-sonnet-4-5"
ANTHROPIC_API_KEY="sk-ant-..."

Then uv run main.py (no proxy needed).

Setup

Step 1: Configure the environment variables

1. Create or edit the .env file in the project root and verify that the following variables are set correctly:

ANTHROPIC_API_KEY=""  # Enter your Anthropic API secret key

Step 2: Install dependencies

Option 1: Setup with uv (Recommended)

uv is a fast Python package installer and resolver.

1. Install uv, if not already installed:

pip install uv

2. Create and activate a virtual environment:

uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install dependencies:

uv pip install -e .

4. Run the project

uv run main.py

Option 2: Setup without uv

1. Create and activate a virtual environment:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

2. Install dependencies:

pip install anthropic python-dotenv prompt-toolkit "mcp[cli]==1.8.0"

3. Run the project

python main.py

Usage

Basic Interaction

Simply type your message and press Enter to chat with the model.

Document Retrieval

Use the @ symbol followed by a document ID to include document content in your query:

> Tell me about @deposition.md

Commands

Use the / prefix to execute commands defined in the MCP server:

> /summarize deposition.md

Commands will auto-complete when you press Tab.

Development

Adding New Documents

Edit the mcp_server.py file to add new documents to the docs dictionary.

Implementing MCP Features

To fully implement the MCP features:

1. Complete the TODOs in mcp_server.py
2. Implement the missing functionality in mcp_client.py

Linting and Typing Check

There are no lint or type checks implemented.