MCP Agent System

Production-grade MCP Server with Agent Tool Orchestration — a Model Context Protocol implementation featuring 6 sandboxed tools, an agent orchestration engine, and an evaluation suite with CI/CD quality gates.

---

Table of Contents

• Overview
• Architecture
• Features
• Project Structure
• Quick Start
• API Endpoints
• Tools
• Evaluation Suite
• Configuration
• Docker Deployment
• CI/CD Pipeline
• Testing
• Tech Stack
• License

---

Overview

MCP Agent System is a production-grade implementation of the Model Context Protocol (MCP) with a built-in agent orchestration engine. It bridges LLM agents and executable tools through a standardized protocol layer, providing a safe and observable environment for autonomous task completion.

The system operates in two modes:

• Stub mode (default): Deterministic, offline, CI-friendly. Tool selection uses keyword matching — no external API calls or LLM dependencies.
• Real mode: LLM-powered agent with real tool execution (configurable via environment variables).

---

Architecture

The system is organized into four core layers:

┌─────────────────────────────────────────────────────────────────┐
│                     FastAPI Dashboard Layer                      │
│        (REST API + real-time tool call monitoring)               │
├─────────────────────────────────────────────────────────────────┤
│                   Agent Orchestration Engine                     │
│   Plan ──► Tool Selection ──► Execution ──► Verification         │
│        (StubPlanner keyword matching / LLM mode)                 │
├─────────────────────────────────────────────────────────────────┤
│                    MCP Protocol Layer                            │
│     Tools (actions)  │  Resources (context)  │  Prompts          │
│        (server.py — registration, invocation, lifecycle)         │
├─────────────────────────────────────────────────────────────────┤
│                    Tool Implementation Layer                     │
│  ┌──────────┬────────────┬───────────┬────────────┬───────────┐ │
│  │Calculator│Code Executor│Web Searcher│File Manager│Data Analyzer│ │
│  └──────────┴────────────┴───────────┴────────────┴───────────┘ │
│  ┌──────────┐                                                   │
│  │Git Helper│   + AST-validated sandbox safety                  │
│  └──────────┘                                                   │
├─────────────────────────────────────────────────────────────────┤
│                    Evaluation Suite                              │
│  Tool Selection Accuracy │ Task Completion │ Quality Score      │
│  Pass-Rate Gate │ Cost & Latency Tracking                        │
└─────────────────────────────────────────────────────────────────┘

MCP Protocol Layer

Implements the three MCP primitives:

• Tools — Executable actions the agent can invoke (POST-like semantics). Each tool has input/output JSON schemas and a safety flag.
• Resources — Read-only context data the agent can reference (GET-like semantics), identified by URI.
• Prompts — Reusable prompt templates with parameterized arguments.

6 Tools

Six fully sandboxed tools covering common agent task domains: calculation, code execution, web search, file management, data analysis, and Git operations.

Agent Engine

Follows a Plan → Execute → Verify loop:

1. Plan — Analyze the task and select the appropriate tool via keyword scoring (stub) or LLM reasoning (real mode).
2. Execute — Call the selected tool with inferred arguments through the MCP server.
3. Verify — Check whether the result satisfies the task; retry with a fallback strategy if needed.

Safety limits enforced: max iterations per task, max total tool calls, and per-tool execution timeout.

Evaluation Suite

Automated evaluation measuring tool selection accuracy, task completion rate, quality scores, latency, and cost — with a configurable pass-rate threshold that gates CI/CD.

---

Features

• MCP Protocol Compliant — Full implementation of Tools, Resources, and Prompts primitives with registration, listing, and invocation.
• 6 Sandboxed Tools — Calculator, code executor, web searcher, file manager, data analyzer, and Git helper.
• AST-Validated Safety — Code execution and calculation tools validate the AST before execution, blocking imports, dunder access, and dangerous builtins.
• Agent Orchestration — Plan → Execute → Verify loop with deterministic stub planner and safety limits.
• Dual-Mode Operation — Stub mode for CI-friendly deterministic runs; real mode for LLM-powered autonomy.
• Evaluation Suite — Automated quality gates with tool selection accuracy, task completion rate, and quality scoring.
• Cost & Latency Tracking — Every tool call records latency and estimated cost for observability.
• Virtual Environments — File manager uses an in-memory virtual filesystem; Git helper operates on a virtual repository — no host system access.
• FastAPI Dashboard — REST API for server stats, tool listing, and real-time tool call monitoring.
• Docker Ready — Production Dockerfile with slim Python image.
• CI/CD Integrated — GitHub Actions workflow with ruff linting, pytest, and eval gate enforcement.

---

Project Structure

mcp-agent-system/
├── app/
│   ├── __init__.py              # Package metadata
│   ├── config.py                # Pydantic settings (env-based configuration)
│   ├── models.py                # Pydantic data models (MCP + Agent + Eval)
│   ├── server/
│   │   ├── __init__.py
│   │   └── server.py            # MCPServer — protocol layer implementation
│   ├── agent/
│   │   ├── __init__.py
│   │   └── engine.py            # Agent + StubPlanner — orchestration engine
│   └── tools/
│       ├── __init__.py
│       ├── registry.py          # Tool registration with the MCP server
│       ├── calculator.py        # AST-validated math expression evaluator
│       ├── code_executor.py     # Sandboxed Python code executor
│       ├── web_searcher.py      # Simulated web search with knowledge base
│       ├── file_manager.py      # Virtual in-memory filesystem manager
│       ├── data_analyzer.py     # CSV data analysis (stats/filter/sort/aggregate)
│       └── git_helper.py        # Virtual Git repository operations
├── eval/                        # Evaluation suite and task datasets
├── scripts/                     # CLI entry points (run_eval, etc.)
├── tests/                       # pytest test suite
├── .github/
│   └── workflows/
│       └── ci.yml               # CI workflow (lint + test + eval gate)
├── pyproject.toml               # Build config, ruff & pytest settings
├── requirements.txt             # Production dependencies
├── requirements-dev.txt         # Development dependencies
├── Dockerfile                   # Container build definition
├── .dockerignore
├── .gitignore
└── README.md

---

Quick Start

Prerequisites

• Python 3.10 or higher
• pip (or your preferred package manager)

Installation

# Clone the repository
git clone https://github.com/your-username/mcp-agent-system.git
cd mcp-agent-system

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install production dependencies
pip install -r requirements.txt

# Or install with development dependencies
pip install -r requirements-dev.txt

Run the Server

# Start the FastAPI server (default: 0.0.0.0:8000)
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

# Or run with custom configuration via environment variables
TRANSPORT=sse AGENT_MODE=stub PORT=8000 uvicorn app.main:app --host 0.0.0.0 --port 8000

The server will be available at http://localhost:8000. Interactive API docs are available at http://localhost:8000/docs (Swagger UI).

Run the CLI

# Run the evaluation suite with default settings
python -m scripts.run_eval

# Run with custom parameters
python -m scripts.run_eval --max-tasks 30 --threshold 0.7

---

API Endpoints

Method	Endpoint	Description
GET	/	Health check and server info.
GET	/tools	List all registered MCP tools with their schemas.
POST	/tools/call	Invoke a tool by name with arguments.
GET	/resources	List all registered MCP resources.
GET	/resources/{uri}	Read a resource by URI.
GET	/prompts	List all registered MCP prompt templates.
GET	/prompts/{name}	Render a prompt template with arguments.
POST	/agent/run	Submit a task to the agent orchestration engine.
GET	/stats	Get server statistics (call count, success rate, etc.).
GET	/health	Liveness probe for container orchestration.

---

Tools

Tool	Description	Key Operations / Features	Safety
calculator	Evaluate mathematical expressions safely.	+, -, *, /, **, %, //, abs, round, min, max, pi, e	AST-validated; blocks non-math nodes
code_executor	Execute Python code in a sandboxed environment.	Arithmetic, string ops, loops, functions, comprehensions, lambdas	AST-validated; blocks imports, dunder access, exec/eval/open
web_searcher	Search the web for information (simulated in stub mode).	Keyword-matched results from a built-in knowledge base	Read-only; no network in stub mode
file_manager	Manage files in a virtual in-memory filesystem.	read, write, list, delete, exists	Virtual FS; blocks directory traversal (..)
data_analyzer	Analyze CSV-formatted data.	stats, filter, sort, aggregate (count/sum/avg/min/max)	Read-only computation; no side effects
git_helper	Perform Git operations on a virtual repository.	status, log, diff, branch, commit	Virtual repo; no host Git access

---

Evaluation Suite

The evaluation suite runs a set of predefined tasks through the agent and measures performance across multiple dimensions. It serves as a quality gate in CI/CD.

Metrics

Metric	Description	Range
Tool Selection Accuracy	Fraction of tasks where the agent selected the correct (expected_tool) tool.	0.0 – 1.0
Task Completion Rate	Fraction of tasks where the agent produced a valid, non-error answer.	0.0 – 1.0
Quality Score	Per-task quality score based on output correctness and completeness.	0.0 – 1.0
Pass Rate	Fraction of tasks with quality_score >= quality_threshold.	0.0 – 1.0
Avg Latency	Average total latency per task (milliseconds).	ms
Avg Cost	Average estimated cost per task (USD), based on token and tool-call costs.	USD
Avg Tool Calls	Average number of tool calls per task.	count

Quality Gate

The eval gate enforces a minimum pass rate before allowing deployment:

# Pass if pass_rate >= threshold (default: 0.7)
python -m scripts.run_eval --max-tasks 30 --threshold 0.7

If the pass rate falls below the threshold, the command exits with a non-zero status code, failing the CI pipeline.

Recommendation Levels

The evaluation summary produces a recommendation based on the pass rate:

Pass Rate	Recommendation	Action
>= 0.8	production_ready	Safe to deploy.
0.7 – 0.8	needs_work	Deployable but improvements recommended.
< 0.7	investigate	Do not deploy; investigate failures.

---

Configuration

All configuration is handled via environment variables (with .env file support) using pydantic-settings. Sensible offline defaults are provided for stub mode.

Variable	Default	Description
AGENT_MODE	stub	Agent mode: stub (deterministic) or real (LLM-powered).
TRANSPORT	stdio	MCP transport: stdio, sse, or streamable_http.
HOST	0.0.0.0	Server bind host.
PORT	8000	Server bind port.
MAX_ITERATIONS	10	Maximum plan-execute-verify iterations per task.
MAX_TOOL_CALLS	20	Maximum total tool calls per task.
TIMEOUT_SECONDS	30	Per-tool execution timeout in seconds.
ENABLE_SANDBOX	True	Enable AST-validated code execution sandbox.
BLOCKED_IMPORTS	["os","subprocess","shutil","ctypes"]	Python modules blocked in the code executor.
BLOCKED_ATTRIBUTES	["__class__","__bases__",...]	Dunder attributes blocked in the code executor.
COST_PER_1K_INPUT_TOKENS	0.00015	Estimated cost per 1,000 input tokens (USD).
COST_PER_1K_OUTPUT_TOKENS	0.0006	Estimated cost per 1,000 output tokens (USD).
COST_PER_TOOL_CALL	0.001	Estimated overhead cost per tool call (USD).
EVAL_MAX_TASKS	30	Maximum tasks per evaluation run.
QUALITY_THRESHOLD	0.7	Minimum quality score for a task to count as "pass".

Create a .env file in the project root to override defaults:

AGENT_MODE=stub
TRANSPORT=sse
HOST=0.0.0.0
PORT=8000
MAX_ITERATIONS=10
ENABLE_SANDBOX=True
QUALITY_THRESHOLD=0.7

---

Docker Deployment

Build the Image

docker build -t mcp-agent-system:latest .

Run the Container

docker run -d \
  --name mcp-agent \
  -p 8000:8000 \
  -e AGENT_MODE=stub \
  -e TRANSPORT=sse \
  mcp-agent-system:latest

The server will be available at http://localhost:8000.

Verify the Deployment

# Health check
curl http://localhost:8000/health

# List tools
curl http://localhost:8000/tools

# Call a tool
curl -X POST http://localhost:8000/tools/call \
  -H "Content-Type: application/json" \
  -d '{"tool_name": "calculator", "arguments": {"expression": "2 + 2"}}'

The Dockerfile uses a multi-step build with python:3.12-slim and caches the dependency layer separately for faster rebuilds. The .dockerignore file excludes tests, scripts, and cache directories to keep the image lean.

---

CI/CD Pipeline

The project includes a GitHub Actions workflow (.github/workflows/ci.yml) that runs on every push and pull request to the main branch.

Pipeline Stages

Step	Description
Checkout	Clone the repository on ubuntu-latest.
Setup Python	Install Python 3.12 with pip caching enabled.
Install Dependencies	pip install -r requirements-dev.txt (includes test and lint tools).
Lint	ruff check . — enforces code style and imports.
Test	pytest — runs the full test suite with verbose output.
Eval Gate	python -m scripts.run_eval --max-tasks 30 --threshold 0.7 — enforces minimum quality.

All steps must pass for the workflow to succeed. The eval gate ensures that regressions in tool selection accuracy or task completion rate are caught before merge.

Badge

Add the CI status badge to your README (replace your-username with your GitHub username/org):

![CI](https://github.com/your-username/mcp-agent-system/actions/workflows/ci.yml/badge.svg)

---

Testing

Run the Test Suite

# Run all tests
pytest

# Run with coverage
pytest --cov=app --cov-report=term-missing

# Run a specific test file
pytest tests/test_calculator.py -v

Run the Evaluation Suite

# Default evaluation (30 tasks, 0.7 threshold)
python -m scripts.run_eval

# Custom evaluation
python -m scripts.run_eval --max-tasks 50 --threshold 0.8

Linting

# Check code style
ruff check .

# Auto-fix issues
ruff check . --fix

Test Categories

• Unit tests — Individual tool handlers, the MCP server, and the agent engine.
• Integration tests — End-to-end agent task runs through the full plan-execute-verify loop.
• Safety tests — Verify the sandbox blocks dangerous code (imports, dunder access, exec/eval).
• Evaluation tests — Verify the eval suite produces correct metrics and gate decisions.

---

Tech Stack

Category	Technology
Language	Python 3.10+
Web Framework	FastAPI 0.104+
ASGI Server	Uvicorn (with standard extras)
Data Validation	Pydantic 2.0+
Configuration	pydantic-settings 2.0+
Linting	Ruff 0.1+
Testing	pytest 7.0+, pytest-asyncio, httpx
Containerization	Docker (python:3.12-slim)
CI/CD	GitHub Actions
Protocol	Model Context Protocol (MCP)

---

License

This project is licensed under the MIT License. See the LICENSE file for details.