Incident Triage MCP Server

An AI-assisted Incident Triage MCP Server that helps engineers investigate service incidents using semantic log search, error aggregation, RAG-based diagnosis, and runbook recommendations.

The project demonstrates how Model Context Protocol (MCP) can be applied to a real-world operational engineering workflow rather than a generic chatbot.

The server operates against synthetic messaging/CPaaS service logs and exposes four MCP tools that can be invoked from an MCP client or directly from the terminal using the FastMCP CLI.

---

🚀 What This Project Does

Instead of manually searching through hundreds of log lines, an engineer can ask questions such as:

Why is messaging-service failing?

Find logs related to queue backlog.

What's the runbook for a connection timeout?

What is the likely root cause of the messaging-service incident?

The MCP server converts these requests into structured tool calls against the incident data.

The important distinction is that the system does not ask an LLM to guess from nothing.

It combines:

• Deterministic log processing
• Semantic search
• Structured error aggregation
• Retrieval-Augmented Generation (RAG)
• LLM-based incident analysis
• Semantic runbook matching

---

🏗️ Architecture

                         ┌─────────────────────────┐
                         │       MCP Client        │
                         │                         │
                         │  Claude / FastMCP CLI   │
                         └────────────┬────────────┘
                                      │
                                      │ MCP / stdio
                                      ▼
                         ┌─────────────────────────┐
                         │     Incident Triage     │
                         │       MCP Server        │
                         │       server.py         │
                         └────────────┬────────────┘
                                      │
              ┌───────────────────────┼───────────────────────┐
              │                       │                       │
              ▼                       ▼                       ▼
       ┌──────────────┐       ┌──────────────┐       ┌──────────────┐
       │ Log Search   │       │ Error        │       │ Runbook      │
       │              │       │ Summary      │       │ Search       │
       │ Embeddings   │       │ Plain Code   │       │ Embeddings   │
       └──────┬───────┘       └──────────────┘       └──────┬───────┘
              │                                              │
              ▼                                              ▼
       ┌──────────────┐                               ┌──────────────┐
       │ logs.jsonl   │                               │runbooks.json │
       └──────────────┘                               └──────────────┘

                                      │
                                      ▼
                              ┌─────────────────┐
                              │ Incident        │
                              │ Diagnosis       │
                              │                 │
                              │ Retrieve Logs   │
                              │       ↓         │
                              │ Anthropic LLM   │
                              │       ↓         │
                              │ Root Cause +    │
                              │ Next Steps      │
                              └─────────────────┘

---

🔧 MCP Tools

The server exposes four tools.

1. search_logs

Performs semantic search over service logs using sentence-transformers.

Example:

fastmcp call server.py search_logs query="queue backlog"

The query is converted into an embedding and compared with log embeddings.

This allows queries such as:

queue is stuck

to find logs containing concepts such as:

consumer lag
queue backlog

without depending entirely on exact keyword matching.

---

2. get_error_summary

Provides a deterministic aggregation of errors.

Example:

fastmcp call server.py get_error_summary

Example result:

Error summary:

db_deadlock: 38
rate_limit_exceeded: 37
queue_backlog: 37
auth_token_expired: 32
connection_timeout: 24

No LLM is involved here.

This is intentional.

Simple counting does not require AI.

---

3. diagnose_incident

Performs RAG-based incident diagnosis.

Example:

fastmcp call server.py diagnose_incident \
  service="messaging-service" \
  hours=24

The flow is:

Service
   │
   ▼
Retrieve recent errors
   │
   ▼
Build incident context
   │
   ▼
Send context to Anthropic
   │
   ▼
Root cause hypothesis
   │
   ▼
Immediate next steps

The LLM receives retrieved incident evidence rather than being asked to diagnose the problem without context.

If ANTHROPIC_API_KEY is not available, the POC falls back to a raw error-frequency summary.

---

4. suggest_runbook

Uses semantic similarity to find the most relevant operational runbooks.

Example:

fastmcp call server.py suggest_runbook \
  query="connection to provider keeps timing out"

Example:

Downstream Connection Timeout
Provider Auth Token Expired
Provider Rate Limiting

This allows engineers to describe an incident naturally instead of remembering the exact runbook title.

---

🧠 AI vs Traditional Logic

One of the goals of this POC is to demonstrate that not everything needs an LLM.

Capability	Approach	Why
search_logs	Embeddings	Semantic matching
get_error_summary	Python aggregation	No AI required
diagnose_incident	RAG + Anthropic	Converts evidence into a diagnosis
suggest_runbook	Embeddings	Handles different incident phrasing

This creates a practical hybrid architecture:

                 Incident Triage
                       │
          ┌────────────┼────────────┐
          │            │            │
      Deterministic  Semantic      LLM
         Logic       Retrieval    Reasoning
          │            │            │
      Aggregation   Embeddings      RAG

---

📁 Project Structure

incident-mcp/
│
├── server.py
├── requirements.txt
├── README.md
├── .gitignore
│
└── data/
    ├── generate_logs.py
    └── runbooks.json

logs.jsonl is generated locally and should not be committed to Git.

---

🛠️ Technology Stack

• Python 3.12
• FastMCP
• Model Context Protocol
• Sentence Transformers
• all-MiniLM-L6-v2
• PyTorch
• NumPy
• SciPy
• Anthropic API
• python-dotenv
• JSONL
• Synthetic operational logs

---

⚙️ Local Setup

1. Clone the repository

git clone git@github.com:balachoudry-tech/incident_mcp_ai.git
cd incident_mcp_ai

---

2. Create a virtual environment

Python 3.12 is recommended.

python3.12 -m venv .venv

Activate it:

macOS / Linux

source .venv/bin/activate

Windows

.venv\Scripts\activate

---

3. Install dependencies

pip install -r requirements.txt

The dependency versions are pinned to the versions used by the working POC environment.

---

📊 Generate Synthetic Logs

Generate the incident data:

python data/generate_logs.py

This creates:

data/logs.jsonl

The generated logs simulate operational failures across messaging/CPaaS services.

---

🔐 Configure Anthropic

Create a .env file in the project root:

ANTHROPIC_API_KEY=your_api_key_here

The .env file is intentionally excluded from Git.

The application loads the key using python-dotenv.

Never commit your API key.

---

▶️ Run the MCP Server

The server uses stdio transport.

python server.py

When started directly, the server waits for an MCP client.

You may see:

Starting MCP server 'incident-triage'
with transport 'stdio'

This is expected.

The server is not a traditional HTTP API.

---

🧪 Test from Terminal

The POC can be tested without Claude Desktop using the FastMCP CLI.

List available tools

fastmcp list server.py

Expected tools:

search_logs
get_error_summary
diagnose_incident
suggest_runbook

---

Test error summary

fastmcp call server.py get_error_summary

---

Test semantic search

fastmcp call server.py search_logs \
  query="queue backlog"

You can also specify the number of results:

fastmcp call server.py search_logs \
  query="queue backlog" \
  top_k=5

---

Test runbook search

fastmcp call server.py suggest_runbook \
  query="connection to provider keeps timing out"

---

Test incident diagnosis

fastmcp call server.py diagnose_incident \
  service="messaging-service" \
  hours=24

With ANTHROPIC_API_KEY configured, this executes the RAG + Anthropic diagnosis flow.

---

🔄 Incident Diagnosis Flow

A typical diagnosis follows this flow:

1. Engineer asks about a service
             │
             ▼
2. Retrieve recent service errors
             │
             ▼
3. Build incident context
             │
             ▼
4. Send relevant context to LLM
             │
             ▼
5. LLM analyzes relationships
             │
             ▼
6. Generate root-cause hypothesis
             │
             ▼
7. Recommend immediate next steps

For example:

Carrier timeout
       │
       ▼
Delivery retries
       │
       ▼
Rate limiting
       │
       ▼
Retry contention
       │
       ▼
Database deadlocks
       │
       ▼
Queue backlog

The LLM can use these correlated signals to produce a higher-level incident hypothesis.

---

📦 Why No Vector Database?

This POC intentionally does not use a vector database.

The log dataset is small enough that embeddings can be:

Loaded
  ↓
Computed
  ↓
Stored in memory
  ↓
Compared using similarity

For a small proof of concept, this keeps the architecture simple.

A persistent vector index would become useful as the dataset grows.

---

🔒 Security Notes

This is a proof of concept.

The following production concerns are intentionally outside the scope:

• Authentication
• Authorization
• Multi-tenancy
• Secret management
• Tool-level permissions
• Audit logging
• Production observability
• Persistent vector storage
• Real log ingestion
• Production-grade error handling

API keys should never be committed to the repository.

---

🚀 Production Evolution

The POC can later evolve toward:

                    ┌──────────────────────┐
                    │ Real Log Sources     │
                    │                      │
                    │ CloudWatch           │
                    │ Elasticsearch        │
                    │ Loki                 │
                    │ OpenSearch            │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ Log Processing       │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ Persistent Vector DB │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ MCP Server           │
                    │                      │
                    │ Search               │
                    │ Summarize            │
                    │ Diagnose             │
                    │ Runbooks             │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ MCP Client           │
                    │ / Incident Platform  │
                    └──────────────────────┘

Potential future improvements include:

• Real production log sources
• Persistent vector indexes
• Authentication and authorization
• Tool-level access control
• Structured tracing
• Incident correlation IDs
• Confidence scoring
• Better "insufficient evidence" handling
• Audit trails
• Production observability
• Multi-service incident correlation

---

🎯 What This POC Demonstrates

This project demonstrates several practical AI engineering patterns:

MCP

Building operational capabilities as reusable MCP tools.

Semantic Search

Using embeddings to search logs based on meaning rather than exact keywords.

RAG

Retrieving relevant operational evidence before asking an LLM to reason about an incident.

Tool Selection

Using deterministic code where deterministic code is sufficient and AI where reasoning provides additional value.

Operational AI

Applying GenAI to an engineering workflow where the output is grounded in actual incident evidence.

---

⚠️ Disclaimer

The logs and incidents in this repository are synthetic and are intended only for demonstration and learning purposes.

The diagnosis generated by the LLM should not be treated as an authoritative production incident response.

---

📄 License

Add the project's license here if/when one is selected.

---

Author

Built as an engineering-focused MCP/RAG POC demonstrating AI-assisted incident triage for high-throughput messaging/CPaaS systems.