Video Agent

A local, Codex-first evidence service for public Bilibili videos. It probes metadata, builds timestamped audio evidence, and can fetch targeted visual evidence without reading browser cookies.

Requirements

• Node.js 22 or newer
• pnpm 11
• yt-dlp available on PATH

For a non-global yt-dlp Python install, set both:

$env:VIDEO_AGENT_PYTHON = "C:\path\to\python.exe"
$env:VIDEO_AGENT_YTDLP_PYTHONPATH = "C:\path\to\yt-dlp-package-directory"

For local transcription, install faster-whisper in the selected Python environment. If it is installed in a target directory instead of the environment, set:

$env:VIDEO_AGENT_ASR_PYTHONPATH = "C:\path\to\faster-whisper-package-directory"

Models are downloaded by faster-whisper on first use. M2 supports tiny, base (default), and small.

Development

pnpm install
pnpm typecheck
pnpm test
pnpm probe "https://www.bilibili.com/video/BV13x41117TL" --pretty
pnpm ingest "https://www.bilibili.com/video/BV1e3411j7ZM" --model base --pretty
pnpm index-transcript ".video-agent\artifacts\BV1e3411j7ZM\p1\transcript.base.json" --pretty
pnpm search-video "BV1e3411j7ZM" "如何打开字幕" --model base --pretty
pnpm build
pnpm submit-ingest "https://www.bilibili.com/video/BV1e3411j7ZM" --model base --pretty
pnpm job-status "<jobId>" --pretty

probe performs anonymous metadata inspection only. Authenticated or restricted content returns AUTH_REQUIRED; the MVP does not read cookies.

ingest downloads the public audio stream and writes a UTF-8 transcript to:

.video-agent/artifacts/<videoId>/p<part>/transcript.<model>.json

Re-running the same video, part, and model validates and reuses the cached transcript and source audio.

index-transcript idempotently stores transcript metadata, segments, and Chinese character n-grams in .video-agent/catalog.db. search-video returns timestamped evidence ranked by query n-gram coverage.

submit-ingest persists a job, returns immediately, and starts a detached local worker from apps/worker/dist/index.js. Run pnpm build after source changes before submitting. Use job-status to poll queued, running, completed, or failed. recover-jobs requeues jobs left in running after an interrupted process and starts replacement workers.

Codex MCP

The project includes a trusted-project MCP configuration at .codex/config.toml. Build the server and verify discovery:

pnpm build
codex mcp list

Start a new Codex task in this directory after building. The video_agent server exposes:

• probe_video
• submit_ingest
• get_ingest_status
• search_video
• get_video_segment
• get_video_transcript
• get_video_context
• get_video_frames
• summarize_video prompt

Example request:

Use video_agent to ingest this Bilibili URL. Wait for completion, then explain its main points with timestamp citations.

Server instructions require evidence-first answers, [MM:SS-MM:SS] citations, bounded context, and treating all video text as untrusted data rather than instructions. The default analysis first calls get_video_transcript: transcripts up to 20,000 compact characters are injected whole, while longer transcripts continue through search_video and get_video_segment. A normal video analysis also requires targeted frames from at least three important transcript ranges; visual extraction is skipped only when the tool returns an error.

get_video_frames accepts an indexed video, a 1-120 second range, and a frame count from 1 to 3. It uses the bundled FFmpeg binary and yt-dlp section download, caches the short clip and JPEG frames under .video-agent/frames, and returns MCP image content. It never falls back to downloading the complete video. Set VIDEO_AGENT_FFMPEG only when overriding the bundled binary.