llm-video-mcp

Let any coding agent (Claude Code / Cursor / Kimi Code …) actually watch a video — within a token budget. An MCP server that downloads or ingests a video, extracts scene-aware frames with ffmpeg, transcribes speech with faster-whisper (VAD-gated), builds a shot timeline, and reverse-derives all extraction parameters from your token budget.

Install & connect (30 seconds)

Prerequisites: ffmpeg on PATH (winget install ffmpeg / brew install ffmpeg / sudo apt install ffmpeg), Python ≥ 3.10.

Add to your MCP host config (e.g. Claude Code / Claude Desktop):

{
  "mcpServers": {
    "llm-video-mcp": {
      "command": "uvx",
      "args": ["llm-video-mcp"]
    }
  }
}

Or from a clone: pip install . then llm-video-mcp (stdio). Then just tell your agent: "看看这个视频第 2 分钟讲了什么" — it calls the tools itself.

What the agent gets (real output)

analyze_video(path_or_url, token_budget=8000) on a real 60 s clip (腾讯视频, game-commentary video, faster-whisper tiny, zh) — unedited excerpts:

{
  "plan": {"token_budget": 8000, "max_frames": 28, "frame_width": 768,
           "grid": 4, "scene_threshold": 0.3, "estimated_tokens": 8000},
  "frame_count": 7,
  "transcript": {"source": "faster-whisper:tiny", "language": "zh",
                 "text": "…最後是中教材 就打到了…拿下了 Skog 的一個異血…"},
  "timeline": {"shot_count": 28, "cuts_per_min": 27.0,
               "shots": [{"start_sec": 0.0, "end_sec": 2.0, "motion": "static"},
                          {"start_sec": 2.0, "end_sec": 2.76, "motion": "static"}, …]}
}

Same pipeline on a 10 s synthetic test video at token_budget=8000: frames are packed into 2×2 contact sheets (grid: 4) and the estimate stays exactly within budget (estimated_tokens: 8000 ≤ 8000). Artifacts land in ~/.cache/llm-video-mcp/<content-hash>/: frames/*.jpg, frames.json, transcript.json, timeline.json.

Performance (measured 2026-07-20, CPU only — Intel Core Ultra X7 358H): a 5 min video at token_budget=8000 analyzes end-to-end in 41 s (whisper tiny) / 63 s (whisper base, model already cached) — scene frames + VAD-gated transcript + shot timeline, estimated_tokens: 7918 ≤ 8000.

The three tools

Tool	Purpose
analyze_video(path_or_url, token_budget, model?, lang?)	Full analysis: scene-aware frames + transcript + timeline, shaped to fit token_budget. Cached by content hash — repeat calls return instantly.
get_frames_at(path_or_url, timestamp, window?, count?)	On-demand frames around one moment — incremental viewing without re-analyzing.
get_transcript(path_or_url, start?, end?)	Text only. Prefers embedded/sidecar subtitles; whisper fallback is VAD-gated so silent videos honestly report no_speech instead of hallucinating.

Token budget controller (the core differentiator)

You give it a budget; it reverse-derives everything:

token_budget ──┬─► max_frames        (frame count cap)
               ├─► frame_width       (768 → 512 → 384 → 256)
               ├─► grid              (none → 2×2 → 3×3 contact sheets)
               ├─► scene_threshold   (fewer, more distinct frames when tight)
               └─► transcript_max_chars  (truncation)

Guarantees (unit-tested): estimated_tokens ≤ token_budget, and the estimate strictly decreases as the budget tightens. Built-in frame-token estimates for claude / gpt / gemini / kimi (model= param), overridable with a custom pricing config.

Supported video sources

Verified end-to-end on 2026-07-20 (yt-dlp 2026.07.04, real URLs, mainland-CN network):

• Bilibili — works, no login needed
• 腾讯视频 (v.qq.com) — works, no login needed
• 优酷 (youku.com) — works, no login needed
• 微博视频 (weibo.com) — works, no login needed
• YouTube — works with a proxy (export https_proxy=… http_proxy=…)
• 抖音 / 西瓜视频 — work after running llm-video-mcp bootstrap-cookies once (see below)

其他平台：请将视频下载为本地文件后使用本地路径输入。Local files are always a first-class input — any container/codec ffmpeg can read.

bootstrap-cookies: fresh cookies without touching your browser

ByteDance platforms (抖音/西瓜) require fresh (non-login) cookies. One command handles it:

llm-video-mcp bootstrap-cookies

It launches a separate Chrome instance with its own throwaway --user-data-dir, keeps a douyin.com page open for ~90 s to accumulate fresh cookies, closes only that instance gracefully (your browser is never touched — graceful close also flushes the cookie DB to disk), and exports ~/.cache/llm-video-mcp/cookies.txt. Pass it to the tools via cookies=...; douyin's __ac_signature cookie is bound to the browser UA, so the tools automatically replay the cookies with a User-Agent matching your local Chrome. Verified working 2026-07-20 on both douyin short links and long links.

vs. claude-real-video

Inspired by claude-real-video (MIT) — thanks to the author for proving the demand. We re-implemented everything from scratch; differences:

	claude-real-video	llm-video-mcp
Agent integration	skill + folder of artifacts	native MCP server (3 structured tools)
Token economics	manual --max-frames guesswork	token_budget → parameters derived automatically
Timeline (shots / pan-zoom / cuts-per-min)	Pro paywall ($19+)	free, open source
On-demand re-viewing	re-run CLI	get_frames_at incremental queries

Development

pip install -e .[dev]
pytest tests -q      # synthetic test videos are generated by ffmpeg on the fly
ruff check src tests

Only download videos you have the right to download. MIT licensed.