StoryForge — AI Backlog & Requirements Assistant

Turn a plain-English business brief into a structured, Agile-ready product backlog (Epics → User Stories → Acceptance Criteria) using an LLM — then measure it, ground it in real documents, and act on it with an agent.

Built as a hands-on tour of the modern Gen-AI product stack: LLM APIs, structured output, prompt engineering, evals, RAG, agents/tools, and MCP.

System map

flowchart LR
  brief([Plain-English brief]) --> gen
  docs([Company BRD / PRD]) --> rag
  subgraph THREE [3 · Ground]
    rag[Retrieve relevant chunks]
  end
  subgraph ONE [1 · Structure]
    gen[Generate backlog]
  end
  rag -. context .-> gen
  gen --> backlog[(Structured backlog)]
  subgraph TWO [2 · Measure]
    eval[Score quality]
  end
  backlog --> eval
  subgraph FOUR [4 · Act]
    agent[Agent + tools] --> jira[(Jira board)]
  end
  backlog --> agent
  agent -. exposed via .-> mcp{{MCP server}}
  mcp -. any AI client .-> ext([Claude Desktop])

A brief and the company's documents flow in; the app structures a backlog, measures its quality, grounds it in real documents, and acts on it with an agent — reusable by any AI through MCP.

---

What it demonstrates

Each phase adds one core Gen-AI concept, and each was built to be understood by running it. Full write-ups (with plain-English explanations) live in CONCEPTS.md.

Phase	Concept	What it does
1	Structured output, system prompts, output validation	Brief → typed Backlog object (Epics → Stories → Acceptance Criteria), forced into a Pydantic schema
2	Evals, golden datasets, regression testing	Rule-based scorers grade backlog quality against a fixed dataset, so prompt changes can be measured, not guessed
3	RAG, embeddings, vector search, grounding	Retrieves relevant chunks of the company's own documents to ground generation — with a relevance guardrail that refuses out-of-scope questions
4	Tool calling, agent loops, MCP	An agent pushes backlog stories to a (local) Jira board via tool calling, exposed over MCP for any AI client to use

Architecture

The app never talks to a specific LLM vendor directly. It talks to an LLMProvider interface (backend/llm/base.py), and a factory picks the concrete provider from the LLM_PROVIDER env var. Today that's Gemini; adding Claude or OpenAI means writing one new class — nothing else changes. The same seam carries generation, embeddings, and agent tool-calling.

backend/
  core/     Phase 1 — schema (models.py), prompt (story_generator.py), CLI
  evals/    Phase 2 — golden dataset, scorers, runner
  rag/      Phase 3 — chunking, vector store, retriever, Q&A guardrail
  agent/    Phase 4 — tools (local Jira), agent loop, MCP server
  llm/      provider interface + Gemini implementation

---

Setup

# 1. Create and activate a virtual environment
python -m venv venv
venv\Scripts\activate        # Windows PowerShell
# source venv/bin/activate   # macOS/Linux

# 2. Install dependencies
python -m pip install -r requirements.txt

# 3. Add your API key
copy .env.example .env       # then edit .env and paste your Gemini key

Get a free Gemini API key at https://aistudio.google.com/apikey

GEMINI_MODEL accepts a comma-separated fallback chain — the app tries each model in order and fails over if one is overloaded (see .env.example).

---

Run it, phase by phase

# Phase 1 — brief → structured backlog
python -m backend.core.cli
python -m backend.core.cli "Build an app for tracking gym workouts"

# Phase 2 — evals
python -m backend.evals.test_scorers      # offline: scorers vs a bad backlog (no API)
python -m backend.evals.runner myrun      # live: generate + score the golden dataset

# Phase 3 — RAG
python -m backend.rag.demo                # index a doc → retrieve → grounded backlog
python -m backend.rag.ask "How long before my appointment can I cancel?"

# Phase 4 — agent + tools
python -m backend.agent.tools.jira_demo   # the tool alone (no AI)
python -m backend.agent.demo              # the agent drives the tool from an instruction

Connect the MCP server to Claude Desktop

Wrap the Jira tools as an MCP server so any MCP client can use them. Add this to your claude_desktop_config.json, then fully restart Claude Desktop:

{
  "mcpServers": {
    "storyforge-jira": {
      "command": "C:\\path\\to\\storyforge\\venv\\Scripts\\python.exe",
      "args": ["-m", "backend.agent.mcp_server"],
      "cwd": "C:\\path\\to\\storyforge",
      "env": { "PYTHONPATH": "C:\\path\\to\\storyforge" }
    }
  }
}

Then ask Claude to "create a Jira ticket for X" and watch it land in jira_board.json.

---

Roadmap

• [x] Phase 1 — Brief → structured backlog (LLM API, system prompts, structured output)
• [x] Phase 2 — Prompt engineering + evals (measure & improve story quality)
• [x] Phase 3 — RAG (ground stories in uploaded BRD/PRD documents) + guardrails
• [x] Phase 4 — Agents + tools + MCP (push to Jira, agentic refinement)
• [ ] Phase 5 — Web UI + multi-user (the demo shell over everything)