Armin Tavassoli - SE333 Final Project

Testing Agent with Decision Table Test Generation and Security Scanning

For my SE333 final project, I built an MCP (Model Context Protocol) server that helps automate testing workflows for Java Maven projects. The main features are decision table-based test generation and security vulnerability scanning, along with coverage analysis and Git automation tools.

What This Does

The agent provides a few key capabilities:

1. Decision Table-Based Test Generation: Generates JUnit test cases by analyzing method logic and creating test matrices that cover different decision paths
2. Security Vulnerability Scanning: Scans Java code for common security issues like SQL injection, XSS, command injection, etc.
3. Coverage Analysis: Finds code that's missing test coverage and suggests what to test
4. Git Automation: Handles common Git workflows like staging, committing, and creating PRs

Features

Coverage Tools

• find_jacoco_path: Finds where the JaCoCo coverage reports are located (HTML, XML, or exec files)
• missing_coverage: Looks through JaCoCo XML reports to find uncovered code and shows coverage stats

Test Generation

• generate_decision_table_tests: Creates JUnit tests using decision tables. It:
  • Looks at method signatures and logic
  • Finds decision points (if/else, switch statements, ternary operators)
  • Builds a test matrix that covers:
    • Null inputs
    • Empty inputs
    • Valid inputs
    • Boundary conditions
    • Edge cases
    • Exception scenarios

Security Scanning

• scan_security_vulnerabilities: Scans Java source code for:
  • SQL Injection risks
  • Command Injection vulnerabilities
  • Path Traversal issues
  • Hardcoded secrets/passwords
  • Insecure random number generation
  • Deserialization risks
  • XSS vulnerabilities
  • Each finding is classified by severity (high, medium, low)

Git Tools

• git_status: Shows what's staged, unstaged, and untracked
• git_add_all: Stages everything (skips build artifacts)
• git_commit: Creates commits with messages that include coverage stats
• git_push: Pushes to the remote repo
• git_pull_request: Creates PRs (needs GitHub CLI or you can do it manually)

Setup

What You Need

• Python 3.12 or newer
• Node.js 18+ (LTS version works best)
• VS Code with Chat view
• Java 11+ and Maven 3.6+
• Git and a GitHub account
• uv package manager (install here)

Installation Steps

1. Install uv

   # macOS/Linux
   curl -LsSf https://astral.sh/uv/install.sh | sh
   
   # Windows
   powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
   

2. Set up Python environment

   cd Armin_Tavassoli_SE333_Final_project
   uv init
   uv venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   

3. Install dependencies

   uv sync
   # Or if that doesn't work:
   uv add mcp fastmcp httpx pypdf python-dotenv
   

4. Connect VS Code to the MCP server

   • Start the server in HTTP mode:

     python server.py
     

     The server will start on http://localhost:8001/sse (or port 8000 if MCP_PORT is not set)
   • Note: VS Code's HTTP MCP client has known compatibility issues with FastMCP's SSE transport. For VS Code, you can either:
     • Option A (Recommended for VS Code): Use stdio mode by setting MCP_USE_STDIO=true and configuring VS Code to use local process (see .vscode/settings.json)
     • Option B: Use HTTP mode - the server runs on HTTP as required, but VS Code may have connection issues. HTTP mode works well with other MCP clients or for testing with curl.
   • In VS Code, press CTRL+SHIFT+P (or CMD+SHIFT+P on Mac) and search for "MCP: Add Server"
   • Enter the server URL: http://localhost:8001 (or http://localhost:8000 if using default port)
   • Name it something like "SE333 Testing Agent"
   • Make sure the tools show up in the Chat view

5. Enable Auto-Approve

   • Press CTRL+SHIFT+P and search for "Chat: Settings"
   • Turn on Auto-Approve
   • Check that all tools are highlighted

6. Set up the Maven project

   cd codebase
   mvn clean install
   mvn test jacoco:report
   

How to Use

Basic Workflow

1. Check coverage

   • Use find_jacoco_path to find the reports
   • Use missing_coverage to see what's not covered

2. Generate tests

   • Use generate_decision_table_tests with a class and method name
   • Example: generate_decision_table_tests(class_name="org.apache.commons.lang3.StringUtils", method_name="isEmpty")

3. Scan for security issues

   • Use scan_security_vulnerabilities to check the codebase
   • Start with high-severity issues

4. Git workflow

   • git_status → git_add_all → git_commit → git_push → git_pull_request

Example: Generating Tests

# In VS Code Chat:
generate_decision_table_tests(
    class_name="org.apache.commons.lang3.StringUtils",
    method_name="isEmpty",
    input_parameters='{"str": "String"}'
)

This will:

1. Look at the method signature and code
2. Find decision points in the logic
3. Generate test cases
4. Give you JUnit test code you can save

Example: Security Scan

scan_security_vulnerabilities(
    class_name="org.apache.commons.lang3.StringUtils",
    severity="high"
)

You'll get JSON back with:

• What vulnerabilities were found
• Where they are (file and line)
• How severe they are
• Code snippets
• How to fix them

Project Structure

Armin_Tavassoli_SE333_Final_project/
├── codebase/                     # Apache Commons Lang3 Maven project
│   ├── pom.xml                   # Maven config with JaCoCo
│   ├── src/
│   │   ├── main/java/            # Source code
│   │   └── test/java/            # Test code
│   └── target/
│       └── site/jacoco/          # Coverage reports
├── .github/
│   └── prompts/
│       └── tester.prompt.md     # Agent prompt config
├── server.py                     # MCP server code
├── pyproject.toml                # Python dependencies
├── README.md                     # This file
├── demo/                         # Demo materials
│   └── final_presentation.mp4   # Video (if applicable)
├── report/                       # Written report
│   └── reflection.pdf           # LaTeX reflection report
└── docs/                         # Extra docs

Tool Reference

find_jacoco_path

Finds where the JaCoCo coverage reports are in the Maven project.

Inputs: None

Returns: Path to the report file, or an error message if reports haven't been generated yet

Example:

find_jacoco_path()

---

missing_coverage

Looks at JaCoCo XML reports to find code that isn't covered by tests.

Inputs:

• class_name (optional): Specific class to check

Returns: JSON with:

• Coverage summary (lines, branches, methods, classes)
• List of classes with low coverage
• Suggestions for what to test

Example:

missing_coverage(class_name="org.apache.commons.lang3.StringUtils")

---

generate_decision_table_tests

Generates JUnit test cases using decision tables.

Inputs:

• class_name (required): Full class name like org.apache.commons.lang3.StringUtils
• method_name (required): Method to test
• input_parameters (optional): JSON describing the parameters

Returns: JUnit test code as a string

Example:

generate_decision_table_tests(
    class_name="org.apache.commons.lang3.StringUtils",
    method_name="isEmpty"
)

---

scan_security_vulnerabilities

Scans Java code for security issues.

Inputs:

• class_name (optional): Specific class to scan
• severity (optional): Filter by "all", "high", "medium", or "low"

Returns: JSON with:

• Total count of vulnerabilities
• Breakdown by severity
• Details for each one:
  • File and line number
  • Type of vulnerability
  • Severity
  • Description
  • Code snippet
  • How to fix it

Example:

scan_security_vulnerabilities(severity="high")

---

Git Tools

git_status

Shows what files are staged, unstaged, or untracked.

git_add_all

Stages all changes (build artifacts are excluded).

git_commit(message)

Creates a commit. The message will include coverage stats if available.

git_push(remote, branch)

Pushes commits to the remote repo.

git_pull_request(base, title, body)

Creates a pull request. You'll need GitHub CLI installed, or you can create it manually.

Troubleshooting

Server Won't Start

If python server.py fails:

1. Check Python version: python --version (needs 3.12+)
2. Make sure dependencies are installed: uv sync
3. Check that the virtual environment is activated
4. See if port 8000 is already in use

Can't Find JaCoCo Reports

If find_jacoco_path says reports aren't found:

1. Run the tests first: cd codebase && mvn clean test
2. Generate the reports: mvn jacoco:report
3. Check that the directory exists: ls codebase/target/site/jacoco/

VS Code Won't Connect

If tools don't show up in Chat:

1. Make sure the server is actually running (check the terminal)
2. Double-check the server URL in VS Code settings
3. Try restarting VS Code
4. Make sure Auto-Approve is enabled
5. Check the server logs for any errors

Test Generation Fails

If generate_decision_table_tests gives an error:

1. Make sure the class name is fully qualified (e.g., org.apache.commons.lang3.StringUtils)
2. Check that the method name is spelled correctly
3. Verify the source file exists in codebase/src/main/java/
4. Make sure the file is readable

Security Scan Finds Nothing

If scan_security_vulnerabilities returns empty results:

1. This might be fine if the code is actually secure
2. Try scanning a specific class: scan_security_vulnerabilities(class_name="...")
3. Try severity="all" to see everything
4. Make sure Java files are in codebase/src/main/java/

Git Commands Fail

If Git tools return errors:

1. Make sure Git is initialized: git status
2. Check that your Git credentials are set up
3. For git_pull_request: Install GitHub CLI (gh) or create the PR manually
4. Make sure you have write access to the repository

Development Notes

Adding New Tools

If you want to add a new MCP tool:

1. Create a function with the @mcp.tool() decorator
2. Write a good docstring
3. Add the tool name to the list in tester.prompt.md
4. Update this README

Testing

To test the agent:

1. Start the server: python server.py
2. In VS Code Chat, try each tool one at a time
3. Check that the outputs look right
4. Try some invalid inputs to test error handling

Tracking Results

Coverage Metrics

To track coverage improvements, use JaCoCo reports. After running mvn test jacoco:report, you can:

1. View HTML report: Open codebase/target/site/jacoco/index.html in a browser
2. Use the MCP tools:
   • find_jacoco_path to locate reports
   • missing_coverage to get detailed statistics

The reports show:

• Line coverage percentage
• Branch coverage percentage
• Method coverage percentage
• Class coverage percentage

For Presentation: Compare before/after coverage by:

• Running initial coverage: cd codebase && mvn clean test jacoco:report
• Generating tests using the agent
• Running coverage again: mvn test jacoco:report
• Comparing the metrics

Security Metrics

Track security improvements:

• Total vulnerabilities found (use scan_security_vulnerabilities)
• Breakdown by severity (high/medium/low)
• Remediation progress
• Files affected

Git Commit History

If you use the Git automation tools, you can track improvements through commit history:

• Each commit includes coverage statistics
• Review commit messages to see progress over time
• Use git log to see the improvement timeline

Future Ideas

Some things I'd like to add later:

• Integration with mutation testing (PIT)
• Support for other languages besides Java
• Better test case prioritization
• Automated test refactoring
• CI/CD integration
• Real-time coverage monitoring
• More advanced security pattern detection

License

This is my SE333 coursework project at DePaul University.

The codebase (Apache Commons Lang3) uses the Apache License 2.0.

Contact

Student: Armin Tavassoli
Course: SE333 - Software Agents
Institution: DePaul University