livekit-agent-simulator

<div align="center"> <img src="lk_sim_illustration.webp" alt="lks — livekit-agent-simulator: black-box LiveKit agent tests with WebRTC, Inbound, Outbound" width="720"> </div>

<div align="center">

</div>

Dial any LiveKit voice agent with an AI simulated caller — WebRTC room, inbound SIP, or outbound SIP — and keep a full forensic log.
Standalone MCP server + CLI (lks, alias: lk-sim). Black-box testing: no imports from the agent under test, no edits to its code or .env.

<div align="center"> <h3>Quick Install</h3>

curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

</div>

Install via coding agent (copy-paste)

Paste into Claude Code, Cursor, Codex, AmpCode, Windsurf, or any coding agent from the repo you want to test:

Install and configure livekit-agent-simulator (CLI: lks) for this project by following the instructions here:
https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md

Target project root is this workspace. Use absolute --root paths. Install the portable CLI if missing, run lks init, help fill .agent-sim/config.yaml from my local env or ask me for LiveKit + Gemini + agent_name, ensure .agent-sim is gitignored, run preflight, and stop before execute if the voice agent worker is not running. Do not edit agent application source outside .agent-sim/.

Same idea, one line:

Install and configure livekit-agent-simulator by following: https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md

---

TL;DR

The Problem

Voice agents fail in ways unit tests never see:

Gap	What you miss
No real caller	Scripts that never interrupt, stall, or switch language
Chat-only evals	No room events, audio timing, or tool spans
Manual QA calls	Not CI-reproducible, no structured PassCriteria
Agent-coupled harnesses	Tests break when you refactor the worker

The Solution

livekit-agent-simulator drives a Gemini Live persona from scenario JSONL over one of three transport modes (Caller.mode), observes transcripts / tools / flow / room events, and writes a timestamped report you can play back.

Surface	What you get
lks CLI	init → preflight → execute → report → web
MCP server	Same ops for Claude Code, Cursor, Codex, …
Transport modes	webrtc_sim · inbound_sip · outbound_human_pickup · outbound_sim_callee (optional agent_dials)
Reports	events.jsonl, timeline.md, summary.json, optional stereo WAV
Judge	Optional LLM PassCriteria scoring

Why Use lks?

Feature	What it does
Black-box dispatch	Only needs agent_name + LiveKit creds
5 transport modes	WebRTC · inbound SIP · outbound human pickup · outbound sim callee · agent_dials
Scenario JSONL	Persona, Caller, Telephony, Execute, Script, PassCriteria, Dispatch
Forensic log	Per-turn events in SQLite + reports/<run-id>/
Report player	Local web UI: audio + transcript sync
CLI ↔ MCP parity	One ops layer — no duplicate run paths
Portable packs	Download installer; no uv/pip required for users

---

Quick Example

# Install once
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

# In the repo you want to test (agent worker must already be running)
lks init --root /path/to/target
# edit /path/to/target/.agent-sim/config.yaml  (LiveKit + Gemini keys, agent_name)

lks preflight --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks report <run-id> --root /path/to/target
lks web --root /path/to/target          # Ctrl+C to stop

---

Design Philosophy

1. The agent under test is a black box.
   We never import or patch target application code. Dispatch metadata is opaque JSON.

2. Generic core, target-owned config.
   Language, timezone, topics, and business strings belong in the target’s .agent-sim/ — not hardcoded in the package.

3. One ops layer for CLI and MCP.
   execute_* validates then runs. No “run vs execute” forks.

4. Forensics over vibes.
   Every run produces structured events you can compare, log, and play back.

5. CI-friendly gates.
   Hard fails on status / assert / script; optional strict judge for softer LLM scoring.

---

How It Works

1. Read <target>/.agent-sim/config.yaml
2. Pick SimLeg from scenario Caller.mode (webrtc_sim | inbound_sip | outbound_human_pickup | outbound_sim_callee | agent_dials)
3. Connect leg → LiveKit room(s) / SIP hairpin as needed; Gemini stays WebRTC in the sim room
4. Bridge audio; observe transcripts, tools, timing, interruptions
5. Write reports/<run-id>/ + runs.sqlite
6. Optional LLM judge vs PassCriteria

                    Caller.mode (scenario)
         ┌───────────────┬────────────────┬──────────────────┬────────────────────┐
         │  webrtc_sim   │  inbound_sip   │   outbound_human_pickup   │ outbound_sim_callee│
         │  room audio   │  sim dials DID │ human answers → │  Gemini SIP callee │
         │               │                │ Gemini colocated│  (2-room hairpin)  │
         └───────┬───────┴────────┬───────┴────────┬─────────┴─────────┬──────────┘
                 │                │                │                   │
                 └────────────────┼────────────────┼───────────────────┘
                                  ▼
                    ┌──────────────────────────┐
                    │  Gemini Live persona     │
                    │  + LiveKit agent (black  │
                    │    box under test)       │
                    └────────────┬─────────────┘
                                 │ observe
                                 ▼
                    reports/<run-id>/ · runs.sqlite · judge

Mode details and config: docs/telephony.md. Templates: inbound-caller-sim, outbound-human-pickup, outbound-callee-sim.

---

How lks Compares

Approach	Real room	AI caller	Forensic log	MCP	Black-box
Manual phone QA	✅	❌	❌	❌	✅
Unit / mock STT	❌	❌	Partial	❌	❌
In-repo agent tests	⚠️	⚠️	Varies	❌	Often coupled
lks	✅ LiveKit	✅ Gemini Live	✅ Full	✅	✅

When to use lks:

• Regression suites for LiveKit voice agents
• Agent-driven CI / coding-agent workflows (MCP)
• Debugging turn-taking, tools, and silence without reading agent source

When it might not be ideal:

• Pure text chatbots with no LiveKit room
• Offline environments without LiveKit + Gemini API access

---

Installation

Quick install (recommended)

Download only — no uv/pip/build on your machine. CI ships a portable pack (embedded Python + deps + report player).

# macOS / Linux
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

# Windows PowerShell
irm "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.ps1" -OutFile "$env:TEMP\lk-sim-install.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\lk-sim-install.ps1" -Verify

Also available from a release asset:

curl -fsSL "https://github.com/quangdang46/livekit-agent-simulator/releases/download/v0.1.0/install.sh" \
  | bash -s -- --verify

Flag	Purpose
--verify	Checksum verification
--ref v0.1.0	Pin release tag
--no-mcp	Skip MCP registration into coding tools
--uninstall	Remove install

By default the installer registers the MCP server livekit-agent-simulator (lks mcp) into detected tools: Claude Code, Cursor, Cline, Windsurf, VS Code Copilot, Gemini CLI, Amazon Q, OpenCode, Codex, Warp.

Agent-oriented install playbook (long form): docs/guide/installation.md
Raw URL for paste into agents:
https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md

From source (maintainers / contributors)

git clone https://github.com/quangdang46/livekit-agent-simulator.git
cd livekit-agent-simulator
uv sync --extra dev
uv run lks --help

Requires Python 3.10–3.13.

Web UI (maintainers)

Users never build this — CI packs web/dist into the wheel as web_static. Source: web/.

pnpm --dir web install
pnpm --dir web build                    # → web/dist/ (attached by Hatch on uv build)
pnpm --dir web dev                      # HMR; proxy /api + /runs → lks web :8765

See web/README.md.

---

Quick Start

# Agent worker must be running and registered with LiveKit
lks guide
lks init --root /path/to/target
# fill .agent-sim/config.yaml

lks preflight --root /path/to/target
lks scenario-init smoke-hello --root /path/to/target   # if needed
lks validate smoke-hello --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks runs --root /path/to/target
lks report <run-id> --root /path/to/target
lks web --root /path/to/target

Minimal scenario (smoke-hello)

{"apiVersion":"agent-sim/v1","kind":"Scenario","metadata":{"id":"smoke-hello","locale":"en-US","tags":["smoke"]}}
{"kind":"Persona","spec":{"name":"Alex","brief":"First-time caller; confirm you reached the right place, then end politely.","goals":["Hear the agent","Say you will call back"],"style":"polite, brief"}}
{"kind":"Execute","spec":{"max_turns":2,"timeout_s":90,"first_speaker":"user"}}
{"kind":"PassCriteria","spec":{"criteria":["The agent responded to the caller","The agent responded in the caller's language"]}}

Optional multi-judge PassCriteria: judges[] + mode (all | majority | any). Assert highlights (tool_order, constraint_respected, recovery/latency): lks guide.

Full-line // comments in scaffolded JSONL are guides — runtime ignores them.

---

Configuration

Target-only data lives under <target>/.agent-sim/ (gitignored). Created by init.

Section	Required	Purpose
livekit.url	yes	wss://… LiveKit Cloud or self-host
livekit.api_key / api_secret	yes	Server API credentials
livekit.agent_name	yes	Must match worker dispatch name
livekit.dispatch_metadata	no	Default opaque JSON string for all runs
simulator.google_api_key	yes	Gemini key for sim caller (+ judge)
simulator.voice.model / voice / language	no	Defaults: flash-live, Puck, en-US
judge.model	no	If set + PassCriteria → post-run LLM judge
observe.record_audio	no (default true)	Local stereo WAV (L=sim, R=agent); no Egress
observe.data_topics	no	Empty = all topics
observe.tool_event_patterns	no	Map data payloads → tool start/end/error

See template: templates/config.yaml. Consumer-specific wiring: docs/portability.md.

---

Commands

CLI and MCP share the same public ops (ops.py). Prefer execute (validate then run).

CLI	MCP tool	Purpose
init	init_project	Scaffold .agent-sim/ + gitignore
guide	guide	Setup/ops guide (markdown)
web	web	Local report player
preflight	preflight	Config + LiveKit connectivity
scenarios	list_scenarios	List scenarios/*.jsonl
plugins	list_plugins	Verify plugins
cues	list_cues	Built-in + local PCM cues
validate	validate_scenario	Schema + lint
export	export_scenario	Parsed scenario JSON
scenario-init	init_scenario	Scaffold JSONL with // guides
execute	execute_scenario	Validate then run one scenario
execute-all	execute_scenarios	Batch (ids / tag)
execute-dict	execute_scenario_dict	In-memory scenario dict
status	get_run_status	SQLite run status
log	get_run_log	Filtered events.jsonl
report	get_run_report	Summary + verdict + paths
compare	compare_runs	Diff two runs; --baseline hard-fails on latency/assert / barge-recovery regression
runs	list_runs	Run history
mcp	—	Start MCP server (stdio)

lks execute smoke-hello --root /path/to/target
lks execute-all --tag smoke --root /path/to/target
lks log <run-id> --root /path/to/target
lks compare <run-a> <run-b> --root /path/to/target
lks compare <baseline> <candidate> --baseline --root /path/to/target
lks web --port 8765 --root /path/to/target

Every MCP tool needs project_root except guide.

MCP config examples

Installer writes this when tools are detected. Manual Cursor:

{
  "mcpServers": {
    "livekit-agent-simulator": {
      "command": "lks",
      "args": ["mcp"],
      "env": {}
    }
  }
}

Dev checkout (package not installed globally):

{
  "mcpServers": {
    "livekit-agent-simulator": {
      "command": "uv",
      "args": ["run", "--directory", "/abs/path/livekit-agent-simulator", "lks", "mcp"]
    }
  }
}

Equivalent one-shot entry: lks-mcp (alias: lk-sim-mcp; same process as lks mcp / lk-sim mcp).

---

Architecture

src/livekit_agent_simulator/
├── cli.py / mcp_server.py     # thin surfaces
├── ops.py                     # shared public ops
├── run_orchestrator.py        # room lifecycle + run
├── scenario.py                # JSONL parse / validate
├── config.py                  # .agent-sim/config.yaml
├── preflight.py
├── asserts.py / suite.py      # CI gates
├── gemini/                    # Live caller + judge
├── livekit/                   # room, dispatch, observe
├── audio/ · script/ · plugins/
└── web/                       # report player server

Layer	Role
Target .agent-sim/	Config, scenarios, reports, local plugins/cues
Package templates/	Scaffold defaults + built-in cues
LiveKit	Room, dispatch, data topics, transcription
Gemini Live	Simulated caller voice + optional judge

---

CI / Release

Workflow	Trigger	What it does
CI	PR / push → main	web UI build, pytest (3.10 + 3.12), lks --help
Release	tag v*	pytest → wheel → portable packs (win/linux/mac) → GitHub Release

# Local check
uv sync --extra dev
pnpm --dir web build
uv run pytest -q

# Release (pre-1.0 may force-retag a single 0.1.0)
git tag v0.1.0
git push origin v0.1.0

---

Troubleshooting

preflight fails connectivity

lks preflight --root /path/to/target
# Confirm livekit.url / api_key / api_secret and that the project is reachable.
# Skip API check while editing config:
lks preflight --no-connectivity --root /path/to/target

Agent never joins the room

• Worker process must be running and registered with the same livekit.agent_name.
• Increase livekit.agent_join_timeout_ms if cold start is slow.
• Check dispatch metadata is valid JSON string if your worker requires it.

Gemini / simulator auth errors

Set simulator.google_api_key in .agent-sim/config.yaml. The sim caller uses Gemini Live (gemini-3.1-flash-live-preview by default).

No audio in report player

With observe.record_audio enabled (default true): reports/<run-id>/conversation.wav

lks web --root /path/to/target

MCP tools not listed

lks mcp   # must be what the host launches
# or reinstall without --no-mcp
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

Scenario validation errors

lks validate my-case --root /path/to/target
lks scenario-init my-case --root /path/to/target   # fresh scaffold with // guides

---

Limitations

What lks Doesn't Do (Yet)

• Not an agent framework — it tests agents; it does not implement business tools
• Not offline-first — needs LiveKit + Gemini (or configured backends)
• Not a load generator — one simulated caller per run (batch via execute-all)

Known Limitations

Capability	Current state	Notes
Black-box dispatch	✅	Opaque metadata only
Multi-caller rooms	❌	Single sim participant
Non-Gemini caller backends	⚠️	Gemini Live is the supported path
Pixel-perfect ASR scoring	❌	Use PassCriteria + judge / asserts
Secrets in config	⚠️ Paste in gitignored YAML	Do not commit .agent-sim/

---

FAQ

Does it modify my agent repo?

Only scaffolds .agent-sim/ (gitignored). It does not edit agent source.

CLI vs MCP — which should I use?

Same ops. Use CLI in terminals/CI; MCP inside coding agents. Prefer execute_* over ad-hoc run paths.

How do I pass project-specific dispatch fields?

livekit.dispatch_metadata or scenario Dispatch.spec.metadata as an opaque JSON string. Core does not parse consumer keys. See docs/portability.md.

Can I assert on tool calls?

Yes — Assert.spec.tools, tool_order (required tool.start subsequence), observe.tool_event_patterns, Script/assert plugins, and/or PassCriteria + judge. See docs/plugins.md and lks guide.

Where are reports stored?

<target>/.agent-sim/reports/<run-id>/ plus runs.sqlite under .agent-sim/.

Is the report player separate?

No — lks web serves the prebuilt player from the install pack. Maintainers build from web/.

---

Docs

Doc	When
AGENTS.md	Rules for AI agents working on this package
docs/smoke-test.md	First end-to-end run
docs/portability.md	Consumer dispatch / observe setup
docs/plugins.md	Verify plugins + Python API
docs/telephony.md	SIP modes + outbound_sim_callee preflight
docs/interrupt-scenario-matrix.md	Barge / backchannel / noise authoring
lks guide	On-demand setup/ops guide (Assert, compare --baseline, PassCriteria)

---

About Contributions

Please don't take this the wrong way, but I do not accept outside contributions for any of my projects. I simply don't have the mental bandwidth to review anything, and it's my name on the thing, so I'm responsible for any problems it causes; thus, the risk-reward is highly asymmetric from my perspective. I'd also have to worry about other "stakeholders," which seems unwise for tools I mostly make for myself for free. Feel free to submit issues, and even PRs if you want to illustrate a proposed fix, but know I won't merge them directly. Instead, I'll have Claude or Codex review submissions via gh and independently decide whether and how to address them. Bug reports in particular are welcome. Sorry if this offends, but I want to avoid wasted time and hurt feelings. I understand this isn't in sync with the prevailing open-source ethos that seeks community contributions, but it's the only way I can move at this velocity and keep my sanity.

---

License

MIT

---

<div align="center">

Black-box LiveKit agent tests. Real rooms. Forensic reports.

</div>