E-Commerce Support Agent

A harness-controlled, tool-connected, RAG-grounded customer support agent for a mock e-commerce company. It answers order-status, delivery, account, and policy questions by calling real (mock) tools and a real retriever — never by inventing an answer — and a Python harness, not the model, decides which tool call is actually allowed to run.

Prerequisites

• Python 3.12
• uv for dependency management — if it's not installed yet: pip install uv (or see the link above for other install methods)
• An Anthropic API key

Setup

git clone <repo-url>
cd <the-folder-git-just-created>
uv sync
cp .env.example .env

uv sync creates a .venv in the project folder and installs everything into it automatically. Every command elsewhere in this README is prefixed with uv run, which runs inside that .venv without needing it activated — but if you'd rather activate it directly (e.g. to run python/streamlit without the uv run prefix):

# macOS/Linux
source .venv/bin/activate

# Windows (PowerShell)
.venv\Scripts\Activate.ps1

# Windows (cmd.exe)
.venv\Scripts\activate.bat

Edit .env:

ANTHROPIC_API_KEY=sk-ant-...
CLAUDE_MODEL=claude-sonnet-5   # any model your account can access

The first run downloads the sentence-transformers/multi-qa-MiniLM-L6-cos-v1 embedding model used for policy retrieval — a few seconds, then cached locally.

How to run it

Chat with the agent (start here):

uv run streamlit run app.py

Pick one of the sample customers from the dropdown — their account status is shown, since a couple are deliberately flagged/suspended so you can demo how the agent handles that — and start chatting.

Or run one scenario at a time, no UI:

uv run python -m scripts.test_order_authorization

See Verifying it works for the full list.

Run the MCP server standalone (for an MCP-compatible client, e.g. Claude Desktop):

uv run python -m src.mcp_server

Project structure

app.py                     Streamlit chat UI — login + chat, no business logic of its own

src/
  agent.py                 CustomerSupportAgent — the harness loop + system prompt
  auth_context.py          AuthContext — the authenticated customer_id for a session
  conversation_memory.py   Short-term, in-process message history (last N turns)
  llm.py                   Thin wrapper around the Anthropic Messages API
  tools.py                 Data-access functions: orders, accounts, products
  action_guard.py          request_order_cancellation — ownership/state/confirmation checks
  tool_registry.py         Tool schemas (Claude tool-use format) + execute_tool() — the harness gate
  policy_retriever.py      Loads/chunks/embeds policies/*.md; cosine-similarity search
  policy_qa.py             search_policies (tool-facing) + the grounding filter/threshold
  mcp_server.py            FastMCP server exposing the same tools over stdio
  support_session.py       create_session / SupportSession — the boundary external callers use
  logging_config.py        Console + rotating-file logging setup
  audit_logger.py          Per-turn and per-tool-call log records

data/            Mock orders, accounts, products (JSON)
policies/        9 Markdown policy documents (chunked by "## " heading)
scripts/         Runnable scenarios and boundary checks (see Verifying it works)
logs/            Generated at runtime, gitignored
project_docs/    Deeper detail: architecture, security model, testing map

Why I built the harness this way

The harness decides, not the model. Claude can only propose a tool call (a name and arguments). tool_registry.execute_tool is the only code path that actually runs one, and it always injects authenticated_customer_id from the session's AuthContext — a value Claude never sees as something it can set. An authorization bypass here isn't a prompting problem to patch; there is no code path where the model supplies whose data it's asking for.

Ticket-type scoping. The four ticket categories this project scopes to (order status, delivery issues, refunds, subscription/account questions) map directly onto the existing tool/retrieval surface, rather than needing a separate classification step: order status and delivery both resolve through lookup_order / list_customer_orders; refund questions resolve through search_policies (actual refund issuance — issue_refund — is explicitly out of scope for this project); account questions resolve through check_account_status for current state and search_policies for anything about why a status exists or how to change it. I didn't add an explicit classify-then-route stage because the four categories already have a clean mapping onto tools Claude selects directly — an extra routing layer would be one more place to misclassify without adding real routing power at this scale.

One MCP permission-boundary decision. The in-process agent surface and the MCP surface enforce ownership through the same underlying checks (action_guard.py, tools.py), but they differ in where the customer ID comes from. On the agent surface, AuthContext is set once at session creation and every tool call gets authenticated_customer_id injected by trusted code. On the MCP surface, an external client has no AuthContext to inject from, so customer_id is a plain tool argument instead — and the server logs every call in full but does not itself verify that argument. I kept this trust boundary explicit and visible (see project_docs/02_security_and_authorization.md) rather than either quietly inheriting the same gap on the trusted surface, or building a fake auth layer for a single-process demo with no real identity provider behind it.

Policy retrieval lives behind MCP, for governance and standardization, not just convenience. search_policies is exposed as an ordinary tool — identically from the in-process agent and from mcp_server.py — rather than a Python-internal pre-fetch step. That means any MCP-compatible client, not just this agent, can call the same governed retrieval capability the same way, instead of every consumer re-implementing the embedding/retrieval pipeline itself. The tradeoff this creates — grounding is no longer a forced step — is real, and it's what the next paragraph addresses.

Grounding is a checked guarantee, not just an instruction. search_policies is a tool Claude decides to call — the system prompt requires it before any policy answer, but a model can still skip it. After the tool-use loop produces an answer with no further tool calls, the harness re-runs the same local retriever directly against the raw customer message before returning that answer; if it finds a strong match and search_policies was never called that turn, the harness forces one more loop iteration with a corrective instruction instead of returning the ungrounded answer. This closes the main gap in "the system prompt tells it to call the tool" — grounding no longer depends entirely on the model remembering to.

A leaner retrieval stack instead of Chroma/FAISS. Policy chunks are embedded once at startup and searched with a plain NumPy cosine-similarity scan (policy_retriever.py) rather than a vector database. At the current corpus size (9 policy documents) that's simpler to reason about and just as correct — no server process or index to manage. This is the deliberate ceiling for this version, not a shortcut: a real vector store is the natural next step once the corpus grows large enough that brute-force search stops being the fast path.

Tools

Tool	Purpose	Auth model
search_policies	Semantic search over policy documents	Not customer-scoped; Claude calls it on its own judgment before any policy answer
lookup_order	Status + delivery info for one order	Denies access if the order belongs to another customer
list_customer_orders	All orders for the authenticated customer	Same
check_account_status	Active / flagged / suspended / can_place_orders	Same
check_product_availability	Stock quantity + availability	Not customer-scoped
request_order_cancellation	Two-step guarded cancellation	Ownership + cancellable-state + explicit confirmation, all checked before anything is written to disk

Every tool returns {"success": bool, ...}; failures carry an error_code (ORDER_NOT_FOUND, ORDER_ACCESS_DENIED, ORDER_NOT_CANCELLABLE, AUTHENTICATION_REQUIRED, etc.) so the agent reacts to a stable code, not free-text error parsing.

Logging & observability

There's no logs/ folder in the repo — it's gitignored, and nothing under it ships. It's created automatically the first time the app runs: configure_logging() (src/logging_config.py) runs on first import of audit_logger or policy_retriever, creates logs/ if it doesn't already exist, and attaches handlers for two files, created the same way:

• logs/application.log — human-readable, one line per event (event=name key=value ...): startup, per-query retrieval performance, one line per conversation turn.
• logs/observability.log — full-detail JSON records, one per tool call and one per conversation turn, linked by session_id. This is what makes RAG grounding traceable: every search_policies call is logged with the exact chunk(s) retrieved (source, section, content, relevance score), not just a claim that retrieval happened.

Both files rotate at 5 MB (5 backups kept) so they don't grow unbounded across a long session. Read observability.log with uv run python -m scripts.view_logs (filters: --customer, --tool, --failures, --type, --summary) rather than opening the raw file.

Verifying it works

Script	What it exercises	Needs ANTHROPIC_API_KEY
scripts/test_order_authorization.py	Full agent loop: cross-customer order access is denied	Yes
scripts/test_account_status.py	Full agent loop: check_account_status for active/flagged/suspended accounts	Yes
scripts/test_agent_cancellation.py	Full agent loop: guarded cancellation, with confirmation	Yes
scripts/test_conversation_memory.py	Full agent loop: multi-turn follow-ups resolved from short-term memory	Yes
scripts/test_support_session.py	create_session login validation + SupportSession.ask() response shape	Yes
scripts/test_search_policies_tool.py	Claude calls search_policies unprompted for a policy question, skips it for a pure operational one	Yes
scripts/test_guarded_cancellation.py	action_guard.request_order_cancellation directly, no LLM	No
scripts/test_tool_registry.py	execute_tool() dispatch and error codes, no LLM	No
scripts/test_policy_qa.py	Policy retrieval/grounding in isolation, no LLM	No
scripts/test_mcp_boundaries.py	MCP schema validation rejecting malformed/unexpected arguments, live fastmcp.Client	No
scripts/test_mcp_tools.py	Lists the tools/schemas the MCP server registers	No
scripts/test_mcp_cancellation.py	MCP cancel_order enforces the same ownership/state/confirmation guard as the agent path	No
scripts/test_mcp_search_policies.py	MCP's search_policies, called directly: relevant vs. irrelevant query	No
scripts/view_logs.py	Terminal viewer for logs/observability.log	No

Run any of them with:

uv run python -m scripts.test_order_authorization

Note: test_guarded_cancellation.py, test_agent_cancellation.py, and test_mcp_cancellation.py all cancel a real order in data/orders.json (ORD-1004). Re-running one after the first hits ORDER_ALREADY_CANCELLED instead of the full confirm flow. Reset with git checkout -- data/orders.json.

See project_docs/ for a deeper look at the architecture, the full security/authorization model, and a requirement-by-requirement testing map.

Known limitations

• Long-term memory is not implemented. Short-term, in-conversation memory (ConversationMemory) exists; a prior-ticket-history lookup keyed by customer ID does not. This is a known gap against the project's requirements, not a design choice — noted here rather than left silent.
• Conversation memory is in-process only — no persistence across runs.
• The MCP surface trusts the calling client's customer_id outright. Every MCP call is logged in full, but logging isn't authentication.
• Policy grounding depends on an embedding-similarity threshold tuned against a handful of measured queries, not a systematic eval set. It's accurate for the cases it's been checked against; a differently-phrased edge case could still land on the wrong side of the cutoff.
• No concurrency handling on the JSON data files — fine for a single local demo, not for concurrent writers.