G2 Product Analytics MCP

A portfolio project exploring a practical question:

How do you let an AI agent answer product-analytics questions autonomously without giving it unrestricted SQL access or letting it invent business definitions?

This project builds a small governed analytics layer, exposes it through Model Context Protocol (MCP), and lets an MCP-capable LLM investigate product questions in natural language.

The synthetic dataset intentionally contains a hidden July 2026 deterioration in mobile review completion. The agent is not told the answer. It has to discover the metric, compare periods, segment the change, and stop where the evidence stops.

What the demo proved

In a cold-start test, the agent was asked:

“One of our product metrics changed materially in July 2026. Using only the G2 Product Analytics MCP tools, investigate what happened.”

The agent independently found:

• review_completion_rate was the material outlier.
• Overall completion fell from 66.7% in June to 60.5% in July.
• The decline was concentrated in mobile: 62.7% → 47.9% (-14.8pp).
• Desktop was nearly flat, which made device type a much cleaner signal than country or product category.
• Daily volume was too thin to claim an exact “cliff day.”
• The evidence established association, not causation.

That distinction is the core design goal: LLM reasoning on top of governed evidence, not free-form guessing against raw tables.

---

Architecture

flowchart TD
    U["User<br/>Natural-language business question"]
    A["LLM / Agent<br/>Reasoning + tool selection"]
    MCP["MCP Server<br/>Tools + Resource + Prompt"]
    SEM["Governed metric layer<br/>definitions + dimensions + ownership + freshness"]
    EXE["Analytics execution<br/>Python + DuckDB"]
    DATA["Synthetic product data<br/>CSV tables"]

    U --> A
    A --> MCP
    MCP --> SEM
    MCP --> EXE
    EXE --> DATA
    SEM --> EXE
    EXE --> MCP
    MCP --> A
    A --> U

Mental model

LLM decides. MCP connects. Tools execute. Semantic definitions govern. Data provides evidence.

MCP is not the model, semantic layer, or database. It is the standardized interface through which the agent discovers and invokes capabilities.

---

Why I built it this way

A naive analytics agent might expose:

run_sql(sql: str)

That is convenient, but it also lets the model:

• silently redefine metrics,
• choose the wrong denominator,
• double-count joins,
• query deprecated or sensitive fields,
• create inconsistent answers across users,
• and sound confident even when the underlying question is ambiguous.

This project takes the opposite approach.

The agent receives a deliberately small set of governed analytical primitives:

Tool	Purpose
list_metrics	Discover approved business metrics.
get_metric_definition	Retrieve the governed definition before interpretation.
get_metric	Calculate one approved metric for an inclusive date range.
compare_review_completion_by_dimension	Slice review completion by an approved dimension.
compare_periods	Quantify absolute and relative change across two periods.
diagnose_review_completion_change	Compare device segments across two periods and explicitly warn that association is not causation.

The server also exposes:

• an MCP resource: metrics://catalog
• an MCP prompt: product_analytics_investigation

---

Repository map

g2-product-analytics-mcp/
├── README.md
├── LICENSE
├── pyproject.toml
├── .gitignore
│
├── data/
│   ├── users.csv
│   ├── software_products.csv
│   ├── buyer_sessions.csv
│   ├── search_events.csv
│   ├── comparison_events.csv
│   ├── reviews.csv
│   ├── review_events.csv
│   └── metric_catalog.csv
│
├── scripts/
│   └── generate_dummy_data.py
│
├── src/
│   └── g2_product_analytics_mcp/
│       ├── __init__.py
│       └── server.py
│
├── tests/
│   └── test_server.py
│
├── docs/
│   ├── ARCHITECTURE.md
│   ├── DATA_MODEL.md
│   ├── CODE_WALKTHROUGH.md
│   ├── AGENT_EVALUATION.md
│   └── INTERVIEW_WALKTHROUGH.md
│
└── .github/
    └── workflows/
        └── test.yml

See docs/CODE_WALKTHROUGH.md for a detailed explanation of the Python code and docs/DATA_MODEL.md for every synthetic table.

---

Quick start

1. Install prerequisites

• Python 3.10+
• uv
• Node/npm only if you want MCP Inspector
• An MCP-capable client such as Claude Code if you want the agent demo

2. Clone and install

git clone <YOUR-REPO-URL>
cd g2-product-analytics-mcp
uv sync --extra dev

3. Run tests

uv run pytest -q

4. Inspect the MCP server manually

uv run mcp dev src/g2_product_analytics_mcp/server.py

MCP Inspector should discover the tools, prompt, and metric-catalog resource.

5. Connect Claude Code

From this repository:

claude mcp add g2-product-analytics -- \
  uv run --with mcp==2.0.0 mcp run src/g2_product_analytics_mcp/server.py

Check health:

claude mcp list

Then launch Claude:

claude

A useful cold-start evaluation prompt is:

One of our product metrics changed materially in July 2026.
Using only the G2 Product Analytics MCP tools, investigate what happened.
Identify the metric, quantify the change, determine when the change emerged,
find where it is most concentrated, and explain what the evidence does and
does not support. Do not inspect source code or CSV files.

---

Reproduce the synthetic data

The committed CSVs make the demo immediately runnable.

They can also be regenerated deterministically:

uv run python scripts/generate_dummy_data.py

The generator uses a fixed random seed and intentionally plants one anomaly: mobile review completion deteriorates in July 2026.

This is synthetic portfolio data. It contains no G2 proprietary information.

---

Tests and guardrails

The current unit tests validate two foundational properties:

1. review_completion_rate is a valid ratio with a non-zero denominator.
2. mobile completion in July is lower than mobile completion in June.

The more important evaluation layer is behavioral. See docs/AGENT_EVALUATION.md for prompts covering:

• definition governance,
• unsupported metrics,
• unsupported dimensions,
• ambiguous questions,
• causality traps,
• leading questions,
• cold-start anomaly investigation.

---

What I would productionize next

The portfolio version is intentionally compact:

CSV → DuckDB → governed Python metrics → MCP → LLM agent

A production version would likely add:

• warehouse/dbt models instead of CSVs,
• a formal semantic/metrics layer,
• identity propagation and SSO/OAuth,
• row- and column-level authorization,
• audit logs and tool-call observability,
• metric lineage/versioning,
• freshness/schema monitoring,
• query budgets and timeouts,
• PII policies,
• golden-question eval suites,
• and a self-service UI/Slack surface.

---

Design tradeoff I learned

The project intentionally contains both atomic tools (get_metric) and a more opinionated workflow tool (diagnose_review_completion_change).

During testing, the agent showed it could compose atomic tools into new workflows, including weekly trend analysis, without a dedicated weekly_trend() function.

That suggests the scalable direction is not to hard-code every analyst question as a new MCP tool. It is to expose a small number of trusted primitives with strong semantics and let the agent compose them.

---

Documentation

• Repository guide
• Architecture
• Data model
• Code walkthrough
• Agent evaluation
• Interview walkthrough

License

MIT. Synthetic data only.