agent-regress

agent-regress is a local MCP (Model Context Protocol) server that provides tools for AI coding agents to execute, evaluate, and diff regression tests for LLM agents. It integrates directly into your existing AI coding environment via the Model Context Protocol (MCP), allowing you to run regression tests against your agents whenever you change a prompt or model. To ensure fully trustworthy, verifiable regression detection, the MVP relies exclusively on deterministic code checks (exact match, regex, JSON schema, exit code) rather than delegating pass/fail judgment to an AI.

Prerequisites

• Node.js (v18 or higher recommended)
• An MCP-compatible client (e.g., Claude Code, Cursor, or OpenCode)

Installation

Currently, agent-regress runs from a local repository clone. (NPM publishing is not yet configured).

1. Clone this repository to your local machine.
2. Ensure Node.js (v18+) is installed.
3. Run npm install to install dependencies.
4. Run npm run build to compile the TypeScript source.

Configuration

You must add this server to your MCP client's configuration file.

Note: Only Kilo Code has been verified end-to-end in our testing. The configuration examples below for Claude Code, Cursor, OpenCode, and others are untested and provided based on general documentation. Contributions and corrections are welcome!

For Claude Code

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or the equivalent location on your OS. Add the following to the mcpServers object:

{
  "mcpServers": {
    "agent-regress": {
      "command": "node",
      "args": ["/absolute/path/to/agent-regress/dist/index.js"]
    }
  }
}

For Cursor

Edit ~/.cursor/mcp.json and add the same configuration block.

Writing evals.yaml

Create a file named evals.yaml in your project directory. This file defines the tests your agent must pass. The server reads this file from the current working directory where the MCP server is running.

evals:
  - name: "Check Error Extraction"
    type: "CLI"
    command: "python sentry_agent.py --issue 'TypeError: null reference'"
    pass_criteria:
      type: "deterministic"
      check_type: "regex"
      value: "Priority: (High|Medium)"

  - name: "Check Health Endpoint"
    type: "HTTP"
    command: "http://localhost:3000/health"
    method: "GET"
    pass_criteria:
      type: "deterministic"
      check_type: "json_schema"
      value: '{"type":"object","properties":{"status":{"const":"ok"}}}'

  - name: "Check Apologetic Tone"
    type: "CLI"
    command: "python sentry_agent.py --issue 'API is slow'"
    pass_criteria:
      type: "semantic"
      description: "The response must contain an apology for the inconvenience."

Running Evals

(Note: If you run npx agent-regress directly in a terminal, it will appear to hang. This is expected behavior! It is an MCP stdio server waiting for a client to connect and communicate via stdin/stdout, not a standalone CLI tool.)

Once configured and your evals.yaml file is in place, prompt your AI client:

"Run my agent evals using EVAL_PROMPT.md instructions."

The AI will call the agent-regress MCP tools to execute the tests and generate a report.

Expected Output

The AI will output a markdown table summarizing the results. A regression (a test that previously passed but now fails) will be marked clearly.

Eval Name	Status	Reason	Evaluated By
Check Error Extraction	🔴 (Fail)	Output did not contain "Priority: High"	deterministic_code
Check Health Endpoint	🟢 (Pass)	Output matched JSON schema	deterministic_code
Check Apologetic Tone	⚪ (Skipped)	SKIPPED_SEMANTIC_MVP	host_ai

Regression Detected: Check Error Extraction flipped from 🟢 to 🔴.

Current Limitations

• Semantic Evaluation Not Executed: The MVP does not execute semantic checks. They will be marked as SKIPPED_SEMANTIC_MVP. This is intentional to ensure all reported results are 100% grounded in deterministic code.
• No Severity Scoring: The tool reports raw pass/fail and regressions. It does not assign severity scores or block gates based on severity.