MCP Video Gen

A self-hosted MCP server that can expose local media-generation backends such as ComfyUI and Blender while also providing local media analysis, editing, FFmpeg, HyperFrames, timeline, subtitle, speech, and audio utilities.

The project is designed for Portainer-only deployments: the public repository contains a normal multi-file application, while a single video-mcp.yml Stack acts as a bootstrap loader for immutable GitHub Releases.

Capabilities

• Discover ComfyUI nodes actually registered by /object_info when ComfyUI is available.
• Scan read-only ComfyUI models/ and custom_nodes/ directories when mounted.
• Inspect compatible custom-node source/documentation files.
• Submit arbitrary valid ComfyUI API workflow JSON and inspect queue/history/output state.
• Guard standard ComfyUI LoadImage against accidental HTTP URLs, MCP resource URIs, /files/... paths and MCP cache file_id values.
• Optionally control host-installed Blender through an authenticated bridge for bpy automation, still rendering, animation rendering, and GLB export.
• Import a server-retrievable HTTPS file reference directly into the persistent MCP cache with streaming, SSRF-oriented public-address checks, size limits and optional SHA-256 verification.
• Retain text, one-shot base64 and chunked base64 upload paths as compatibility fallbacks when a native/retrievable file reference is unavailable.
• Return cached files as native MCP ResourceLink references and authenticated HTTP(S) streams, with MCP resource/base64 paths retained as compatibility fallbacks.
• Upload inputs, cache outputs, and retrieve generated image, video, audio, 3D, scene, subtitle, and other files through one canonical file_id contract.
• Transfer cached images directly to ComfyUI as multipart binary data without routing file bytes through the AI/client context or converting them to base64.
• Expose file_transfer_guide so an AI can query the canonical file-routing rules at runtime instead of guessing which identifier belongs to which backend.
• Create and render local HyperFrames projects using HTML/CSS/media.
• Probe, transcode, concatenate, overlay, mux audio, crop, reverse, loop, speed-ramp, and extract frames with FFmpeg.
• Detect silence, black/frozen sections, loudness, interlacing, crop regions, keyframes, and objective SSIM/PSNR differences.
• Build contact sheets/storyboards and perform lightweight frame similarity, motion, duplicate-frame, and best-frame analysis.
• Detect and split scenes with PySceneDetect.
• Create, retime, convert, style, and burn subtitles with pysubs2 + FFmpeg.
• Maintain persistent OpenTimelineIO timelines with tracks, clips, transitions, markers, reordering, inspection, and export.
• Detect beats, tempo, onsets, and pitch with aubio.
• Denoise speech locally with RNNoise.
• Detect speech segments with a small Silero VAD ONNX model.
• Transcribe media, generate subtitles, and obtain word-like timestamps locally with whisper.cpp.
• Optionally synthesize speech with user-supplied Piper voices; Piper is disabled by default.
• Optionally validate Cloudflare Access JWTs at the origin and run a Cloudflare Tunnel sidecar.

This server intentionally does not contain fixed AI-generation workflows, long-term memory, or agent skills. It exposes execution primitives so a client or separate knowledge/skills MCP can decide how workflows should be built.

ComfyUI and Blender are external optional backends. If either backend is disabled, missing, or temporarily unreachable, the MCP itself remains healthy. Network-dependent tools return a model-readable available=false result instead of taking down the server or surfacing a backend-absence tool error.

Architecture

MCP client / AI
   |
   |<---- native file references / streaming ---->
   v
MCP Video Gen + persistent file_id cache
   |
   |---------------- optional ComfyUI API
   |                    |
   |                    +-- installed models
   |                    +-- custom nodes
   |                    +-- image/video/audio generation
   |
   |---------------- optional Blender bridge on VM
   |                    |
   |                    +-- bpy scene creation/editing
   |                    +-- .blend / GLB export
   |                    +-- still / animation rendering
   |
   |---------------- HyperFrames
   |---------------- OpenTimelineIO / subtitles
   |---------------- scene / frame analysis
   |---------------- whisper.cpp / Silero VAD / RNNoise / aubio
   +---------------- FFmpeg

All execution paths exchange files through the same MCP cache.

Portainer deployment

Use video-mcp.yml as the Stack definition.

Optional ComfyUI

Typical ComfyUI connection variables are:

COMFYUI_HOST=host.docker.internal
COMFYUI_PORT=8188
COMFYUI_SCHEME=http

For filesystem discovery, set the host paths when ComfyUI exists:

COMFYUI_MODELS_PATH=/host/path/to/ComfyUI/models
COMFYUI_CUSTOM_NODES_PATH=/host/path/to/ComfyUI/custom_nodes

These path variables are no longer required for MCP startup. The Stack has generic empty-directory fallbacks, so it can start before ComfyUI is installed. If ComfyUI is unreachable, its network tools report that status to the model while local MCP tools continue working.

Optional Blender

Blender is disabled by default:

BLENDER_ENABLED=false
BLENDER_BRIDGE_URL=http://host.docker.internal:9876
BLENDER_BRIDGE_TOKEN=
BLENDER_BRIDGE_TIMEOUT_SEC=7200

The recommended integration runs scripts/blender_bridge.py directly on the VM as a dedicated low-privilege OS account. The container communicates with it over an authenticated local HTTP bridge; Blender itself runs headless on the host. This avoids mounting the host root filesystem or host executables into the MCP container.

After the bridge is installed, configure privately in Portainer:

BLENDER_ENABLED=true
BLENDER_BRIDGE_URL=http://host.docker.internal:9876
BLENDER_BRIDGE_TOKEN=<same long random token used by the host bridge>

See docs/BLENDER_BRIDGE.md for setup, security, systemd hardening, file flow, and examples.

Cloudflare Tunnel

If you use the included Cloudflare Tunnel sidecar, provide:

CLOUDFLARED_TUNNEL_TOKEN=<set privately in Portainer>

Point the remote Tunnel hostname to:

http://video-mcp:8000

The MCP endpoint is:

https://your-public-host.example/mcp

Cloudflare Access / Managed OAuth

The application can verify the Cloudflare Access JWT at the origin. Configure these values privately in Portainer:

CF_ACCESS_VERIFY=true
CF_ACCESS_TEAM_DOMAIN=https://your-team.cloudflareaccess.com
CF_ACCESS_AUD=<Access application audience tag>
PUBLIC_BASE_URL=https://your-public-host.example

PUBLIC_BASE_URL also allows get_cached_file_resource to return the authenticated HTTP(S) /files/{file_id} stream as its preferred file reference. Without it, the tool returns the MCP-native media://cache/{file_id} resource URI instead.

No real domain, audience, tunnel token, bridge token, internal IP, or credential belongs in this public repository.

External backend availability

external_backends_status reports the current state of ComfyUI and Blender. inventory_summary includes the same backend status alongside local capabilities.

When an external backend is unavailable, calls return a structure such as:

{
  "ok": false,
  "available": false,
  "backend": "blender",
  "status": "unavailable",
  "message": "Blender integration is disabled..."
}

This is deliberately different from an MCP server failure: the model learns that one optional execution path is unavailable and can continue with another one.

File transfer and shared cache

Every generated/imported artifact is normalized into the MCP cache and identified by file_id. This is the interchange layer between the AI client, ComfyUI, Blender, FFmpeg, HyperFrames, subtitles, timelines, and audio utilities.

The project rule is simple: do not route binary media through inline/chunked base64 when a native file/resource reference or streaming transfer is available. Never resize, recompress, transcode, or otherwise alter an artifact solely to make it fit through a tool result.

When an AI is uncertain which identifier to pass to which system, it should call:

file_transfer_guide()

The most important rule is that standard ComfyUI LoadImage needs a ComfyUI input filename, not a URL, ResourceLink, /files/... path, or MCP file_id.

Client / AI -> MCP cache — preferred path

If the client exposes a real HTTPS URL that the MCP server itself can retrieve, use:

import_remote_file(uri, filename="", expected_size_bytes=0, expected_sha256="")

The file is streamed directly to disk and promoted into the MCP cache as a new file_id; the binary payload is not placed into the model/tool JSON as base64.

The remote importer requires HTTPS, rejects embedded credentials and non-public DNS results, revalidates redirects, enforces MAX_UPLOAD_MB both from Content-Length and while streaming, and optionally validates expected size and SHA-256.

Optional hardening for deployments with known file providers:

REMOTE_IMPORT_ALLOWED_HOSTS=files.example.com,*.storage.example.com
REMOTE_IMPORT_MAX_REDIRECTS=5
REMOTE_IMPORT_TIMEOUT_SEC=120

An opaque ChatGPT/OpenAI attachment file ID is not automatically retrievable by an arbitrary MCP server. Do not invent a URL. Use import_remote_file only when the client actually exposes a server-retrievable HTTPS reference.

Compatibility imports remain:

cache_text_file
cache_file_base64
file_upload_begin
file_upload_chunk
file_upload_finish
file_upload_abort

Chunked uploads can specify expected byte length and SHA-256 before promotion into the persistent cache. The chunking avoids one giant request, but binary chunks are still base64 and are therefore a fallback when a retrievable reference is unavailable.

MCP cache -> ComfyUI image input

For an image already in the cache, use:

comfy_upload_cached_image(file_id)

The adapter uploads the original cached bytes directly to ComfyUI /upload/image as multipart/form-data. It does not base64-encode the asset and does not send the payload through the model context.

A successful response includes:

workflow_load_image_value

Use that exact returned value in:

LoadImage.inputs.image

Do not put the original HTTPS URL, MCP ResourceLink, /files/... path or file_id into LoadImage. submit_workflow explicitly rejects these common mistakes for standard LoadImage nodes and returns instructions for the correct cache -> ComfyUI upload route.

There is currently no generic standard cache -> ComfyUI adapter for arbitrary audio/video. Introspect and use a media-specific installed node/API rather than feeding it a made-up URL contract.

ComfyUI -> MCP cache

After a workflow finishes and its output filename/subfolder/type are known, use:

cache_output(filename, subfolder, output_type)

The result becomes a normal MCP cache artifact with a new file_id and can be consumed server-side by FFmpeg, Blender, HyperFrames and the other tools.

MCP cache -> client / AI — preferred path

Use:

get_cached_file_resource(file_id)

It returns an MCP ResourceLink with URI, filename, MIME type and size, without embedding the binary payload in the tool result.

When PUBLIC_BASE_URL is set, the URI points to:

<PUBLIC_BASE_URL>/files/<file_id>

The /files/{file_id} route supports GET, HEAD and HTTP byte ranges through Starlette FileResponse, making it the preferred transfer path for large video, audio, images, .blend, GLB and other binary outputs.

Without a public HTTP(S) base URL, the ResourceLink falls back to:

media://cache/<file_id>

The server exposes the matching media://cache/{file_id} MCP resource template. Binary resources/read is a compatibility path rather than the preferred large-file path because binary MCP resource contents are base64 blobs on the wire.

Metadata remains available through:

get_cached_file_info

Compatibility/debug fallbacks remain:

read_cached_file_chunk_base64
get_output_inline_base64

MCP cache <-> Blender and HyperFrames

Blender MCP tools take cache file_id values directly and stream the corresponding files server-side to the host bridge; declared Blender outputs are returned to the MCP cache as new file_id values.

For HyperFrames, use:

hyperframes_import_cached_media(project_id, file_id, destination)

Do not read an artifact through base64 merely to write it into Blender or HyperFrames.

See docs/FILE_HANDOFF.md for the complete routing matrix, resource semantics, authentication notes, security controls and large-file rules.

Advanced local media utilities

The runtime prepares several small local utilities in addition to FFmpeg/HyperFrames. The Python venv contains PySceneDetect, OpenTimelineIO, pysubs2, ONNX Runtime, NumPy, and headless OpenCV. Debian provides the small aubio-tools CLI package. RNNoise and whisper.cpp are built locally from pinned upstream source into the persistent data volume.

Silero VAD, RNNoise, and whisper.cpp model/source artifacts are stored under the persistent data volume. The RNNoise source and model plus the Silero/Whisper model downloads use explicit SHA-256 validation. The default Whisper model is a small quantized model intended for lightweight local transcription; model URLs/hashes and source refs can be overridden through Stack variables.

Relevant variables include:

SILERO_VAD_ENABLED=true
SILERO_VAD_MODEL_URL=<public model URL>
SILERO_VAD_MODEL_SHA256=<expected sha256>

RNNOISE_ENABLED=true
RNNOISE_REF=<pinned upstream commit>
RNNOISE_SOURCE_URL=<public source archive URL>
RNNOISE_SOURCE_SHA256=<expected sha256>
RNNOISE_MODEL_URL=<public model URL>
RNNOISE_MODEL_SHA256=<expected sha256>

WHISPER_CPP_ENABLED=true
WHISPER_CPP_REF=v1.8.6
WHISPER_CPP_BUILD_JOBS=2
WHISPER_MODEL_AUTO_DOWNLOAD=true
WHISPER_MODEL_NAME=tiny-q5_1
WHISPER_MODEL_URL=<public model URL>
WHISPER_MODEL_SHA256=<expected sha256>

The first startup after enabling these utilities can take longer because RNNoise and whisper.cpp are built locally and the selected assets are downloaded. Their resulting builds and models remain in /data, so normal container recreation does not repeat those builds when the persistent volume is retained. The Stack gives the first startup an extended healthcheck grace period for this reason.

Optional Piper TTS

Piper is implemented as an optional runtime and is disabled by default:

PIPER_ENABLED=false
PIPER_PACKAGE_SPEC=piper-tts

When enabled, no voice is downloaded automatically. Voice .onnx and matching configuration files live under /data/piper/voices; they can be imported from the MCP media cache with piper_import_voice_file. This keeps TTS optional because ComfyUI itself can also host audio/TTS workflows.

See THIRD_PARTY.md for third-party licensing notes.

Release selection

The Stack supports:

VIDEO_MCP_VERSION=latest
VIDEO_MCP_CHECK_UPDATES_ON_START=true
VIDEO_MCP_FORCE_REFRESH=false

latest means the highest non-draft, non-prerelease GitHub Release whose tag exactly matches vX.Y.Z. It does not mean main.

You can also pin a release:

VIDEO_MCP_VERSION=v2.6.0

or a commit SHA:

VIDEO_MCP_VERSION=<commit-sha>

When update checking is disabled and a valid /current source exists, startup is completely cache-first. A failed release lookup, download, or archive validation falls back to the last-known-good source whenever one exists.

Persistent volumes

The Stack separates three concerns:

video_mcp_code  -> /opt/video-mcp   versioned source cache + /current
video_mcp_venv  -> /opt/venv        persistent Python virtual environment
video_mcp_data  -> /data             media, timelines, models, local tooling, HyperFrames projects/cache

The application runtime data root defaults to /data. Direct/non-Stack deployments may override it with VIDEO_MCP_DATA_ROOT; importing video_mcp.server or video_mcp.entrypoint does not create the directory. Runtime directories are created only when the application starts.

The Python environment is rebuilt only when requirements.txt changes. Rebuilding clears the contents of the mounted venv directory; it never removes the Docker mount point itself.

Source bootstrap security

Source archives are downloaded from GitHub codeload into a staging directory and validated before extraction. The bootstrap rejects:

• absolute paths;
• .. traversal;
• symbolic links;
• hard links;
• archives with more than one top-level root.

A release receives .mcp-source-ready only after extraction and runtime-contract checks succeed. /current is switched only after that point, so an interrupted or malformed update cannot replace the last-known-good source.

The ComfyUI model and custom-node filesystem mounts are read-only. AI utility source/model downloads use temporary files and SHA-256 verification before replacing cached artifacts. The optional Blender bridge uses bearer-token authentication and only transports declared job inputs/outputs, but arbitrary Blender Python remains powerful and therefore the bridge must be isolated with an unprivileged OS account.

HyperFrames

HyperFrames runs locally in the MCP container and uses the same persistent /data area as the MCP media cache. Browser assets are cached persistently under /data/hyperframes-home.

The default package spec is pinned in the Stack for reproducibility and can be overridden privately:

HYPERFRAMES_NPM_SPEC=hyperframes@0.7.111

HyperFrames skills are intentionally disabled in this execution server (HYPERFRAMES_SKIP_SKILLS=1).

Development

python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt pytest PyYAML
PYTHONPATH=src python -m pytest -q
python scripts/check_public_repo.py

CI checks Python compilation, server/entrypoint imports, tests, YAML parsing, shell/Python helper syntax, Compose rendering, version/changelog consistency, and public-repository secret/private-network guardrails.

Release process

1. Develop on a branch and open a PR.
2. CI must pass.
3. Update VERSION and CHANGELOG.md.
4. Merge to main.
5. CI creates the immutable vX.Y.Z tag and matching stable GitHub Release if it does not already exist.

Application tags/releases are reserved for exact vX.Y.Z names so unrelated model or asset releases cannot affect VIDEO_MCP_VERSION=latest resolution.

License and attribution

Licensed under the Apache License 2.0. See LICENSE.

Redistributions and derivative works must preserve the attribution notice in NOTICE in accordance with the Apache License 2.0. Third-party components retain their own licenses; see THIRD_PARTY.md.