Cleya

<p align="center"> <img src="assets/cover.png" alt="Cleya — creative tools that check their own work" width="100%"> </p>

<h1 align="center">Cleya</h1>

<p align="center"><em>A creative-generation and video toolkit for AI agents. Works with any LLM to make beautiful creative, from statics to 2D animation, then <strong>checks its own work</strong> before handing it over.</em></p>

<p align="center"> <a href="https://github.com/severment/cleya/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/severment/cleya/ci.yml?branch=main&style=flat-square&label=tests&labelColor=14231F&color=17958A"></a> <img alt="license" src="https://img.shields.io/badge/license-Apache--2.0-17958A?style=flat-square&labelColor=14231F"> <img alt="node" src="https://img.shields.io/badge/node-%E2%89%A518-17958A?style=flat-square&labelColor=14231F"> <img alt="MCP" src="https://img.shields.io/badge/MCP-ready-2DD4BF?style=flat-square&labelColor=14231F"> <img alt="surfaces" src="https://img.shields.io/badge/CLI_·MCP·_library-2DD4BF?style=flat-square&labelColor=14231F"> </p>

---

Status: v0.6.0, actively maintained. 834 tests across 75 files, a documented scene grammar with a JSON Schema beside it, and a contract-tested starter library — a change that breaks a starter breaks the build. Issues and pull requests welcome.

---

What it is

67 tools on three surfaces — a CLI, an MCP server, and a TypeScript library — covering the whole path from "I have a brief" to "here is a measured, on-brand MP4":

• Author. Describe an animation as declarative JSON (the scene DSL) instead of hand-writing HTML and timing code. Layers, entrances, idle motion, keyframe tracks, camera moves, tickers, data binding, multi-format output.
• Render. Headless Chrome samples window.CLEYA.seekTo(t) once per frame, so every frame is a pure function of time — deterministic, seekable, resolution-independent — then FFmpeg encodes.
• Measure. Perception instruments read the rendered pixels back: WCAG contrast per text box, optical alignment, dominant-colour palette, brand conformance by ΔE. The agent that made the creative can see whether the creative is actually any good.

That last step is the point. Generation is commodity; the loop is made → measured → corrected, and it runs without a human in it.

The loop, end to end

# 1. start from a guaranteed-valid skeleton
cleya starters --name promo-hero --copy-to ./promo

# 2. render it with your brand tokens
cleya render --scene promo/promo-hero.scene.json --brand brand.json --output promo.mp4

# 3. measure what came out
cleya contrast    --input promo.mp4 --time 3.2                    # WCAG per OCR'd text box
cleya align       --input promo.mp4 --time 3.2                    # near-miss edges, broken rhythm
cleya brand_check --input promo.mp4 --time 3.2 --brand brand.json # 0-100 ΔE conformance

# 4. correct deterministically — refinement as code, not deliberation
cleya edit --scene promo/promo-hero.scene.json --op '{"op":"center","layer":"headline"}' --verify

# 5. promote a good result into a reusable starter
cleya harvest --scene promo/promo-hero.scene.json --brand brand.json \
  --name book-flip-promo --when "Book-brand promo with flipping titles"

Or collapse steps 1–3 into one call:

cleya draft --starter promo-hero --brand brand.json \
  --copy '{"headline":"Ship it {accent:Friday}","tagline":"in one command","stat":42}' \
  --output draft.mp4
# → rendered + contrast-checked + brand-scored, ~15s
# (layer ids and knobs per starter: cleya starters --name promo-hero)

Quick start

Cleya is not published to npm. Install from source:

git clone https://github.com/severment/cleya.git
cd cleya && npm install && npm run build && npm link
cleya doctor          # check FFmpeg, ffprobe, Chrome, Whisper
cleya recipes         # goal → tool-sequence discovery

As an MCP server

{
  "mcpServers": {
    "cleya": { "command": "cleya-mcp" }
  }
}

Every tool is registered as cleya_<tool> with a typed schema. Point an agent at cleya_capabilities first — one call returns the whole machine-readable manifest (options, examples, categories, cost hints), which is how an agent learns the toolkit without guessing.

As a library

import { render, contrast, brandCheck, resolveConfig } from "cleya";

const config = resolveConfig();
const out = await render({ scene: "promo.json", brand: "brand.json", output: "promo.mp4" }, config);
const wcag = await contrast({ input: "promo.mp4", time: 3.2 }, config);

Every tool takes (params, config) and returns Promise<ToolResult<T>>.

Authoring — the scene DSL

A scene is data, not code. The compiler turns it into the deterministic seekTo runtime that the capture backends drive:

{
  "size": [1920, 1080], "duration": 10, "bg": "$palette.bg",
  "layers": [
    { "id": "h1", "type": "text", "text": "MAKE IT MOVE", "at": { "x": 118, "y": 208 },
      "style": { "size": 104, "weight": 900, "color": "$palette.ink", "uppercase": true },
      "in": { "at": 0.45, "dur": 0.75, "from": "rise", "ease": "outCubic" } },
    { "id": "cta", "type": "text", "text": "Get started", "anchor": "bottom-left", "safe": true,
      "style": { "bg": "$palette.accent", "pad": [18, 34], "radius": 999 },
      "in": { "at": 2.1, "dur": 0.5, "from": "pop" },
      "idle": { "kind": "breathe", "amp": 0.02 } }
  ]
}

• --check validates without rendering (~0.05s) and warns on dead air.
• --at <t> renders one frame; --preview writes a labelled contact sheet — this is how an agent "watches" a video without watching it.
• --format og,square,story emits every size from one spec, with per-layer overrides.
• --brand brand.json resolves $brand.* / $palette.* tokens and merges fonts.
• --data / --data-each bind typed values and batch-render one spec into N named outputs.
• cleya scrub opens a film in a self-contained scrub page — timeline with hover-scrub, frame-stepping, loop — with zero rendering, because it plays through the same seekTo contract. With --brand, it also carries a live palette bar: re-tint any brand role and the film updates instantly, then download the edited brand.json to render with. Works for raw --html films too, as long as their colours are authored as var(--cleya-<role>, <hex>).

type: "video" layers put real footage on the same timeline — the runtime seeks currentTime per frame rather than playing, so a cut is still a pure function of scene time.

Full grammar: docs/scene-spec.md · schema: docs/scene.schema.json · craft guidance for ambitious work: docs/craft.md

When the DSL genuinely isn't enough, render --html --runtime gsap is the escape hatch: it injects GSAP with every plugin registered plus window.cleya helpers, and you author a paused timeline that the same seekTo bridge drives.

Skills

skills/ ships agent-readable craft guidance, one directory per lens — read before authoring, not after:

Skill	For
edit-craft	The grammar of cuts: which cut, when to cut, ad pacing, and how each maps onto layer windows
marketing-producer	Campaign work — offer, proof, CTA
social-editor	Vertical, sound-off, first-three-seconds
documentary-editor	Long-form, interview-led
podcast-producer	Multi-speaker audio-first

Perception instruments

The part nothing else in this category ships. All of these read rendered pixels, not source specs.

They work on any video, not just one Cleya rendered. contrast, align, palette, brand_check, review and pacing take --input <file> and measure whatever is in it. Point them at output from another renderer, at footage from an editor, at a file someone sent you. The measuring half of this toolkit is not coupled to the authoring half, so you can adopt it without moving anything you already have. (scrub is the exception — it needs a seekTo film to drive.)

Tool	What it measures
contrast	WCAG ratio per OCR'd text box — strokes split from ground by Otsu, so it measures the text, not the box
align	Infers the frame's alignment systems, then flags only a lone element that almost joins one — two deliberate systems a few px apart are not a defect. Plus broken vertical rhythm
palette	Coverage-weighted dominant colours (ΔE-merged) + luminance stats
brand_check	0–100 brand conformance against a BrandProfile by ΔE, neutrals passing; optional scene font check
review	Render QA — black/empty/frozen frames, text collisions, unsafe margins, tiny text, audio dead air; --platform tiktok|reels|shorts adds UI-occlusion zones
claims	Lints copy against a claims policy — banned phrases, required disclaimer, line length, reading grade
pacing	The film's rhythm: holds, beats, segment lengths, end-card dwell, plus an activity strip. The only instrument that reads the film as a film rather than a frame, and the only way to check "the end card holds for three seconds" without watching it
diff	Where two frames differ (bounding box), by how much (per-channel max), PSNR, and a difference image amplified to the change it actually contains

harvest and edit --verify are instrument-gated: alignment defects block, contrast findings warn. A defective scene cannot silently enter your starter library.

The rest of the toolkit

Run cleya --help for all 67, or cleya capabilities for the machine-readable manifest.

• Author — render, screenshot, card, title, caption, still, tts, capture (scripted real-app screen recording with synthetic cursor, auth state seeding, dev-overlay hiding)
• Pipeline — draft, compose (multi-scene assembly with transitions, music bed, ducking, beat-snap, per-scene VO and content-hashed render cache), social_clip, autosub
• Edit — trim, concat, speed, reformat, audio, overlay, pip, color, lut, effects, shake, upscale, gif, kenburns, layout, multiclip, sync, bg_remove
• Scout — info, scenes, motion, loudness, beat_detect, cut_silence, thumbnails (saliency-ranked), transcribe, detect_text, image_similarity
• Assets — cutout (Vision foreground segmentation to transparency), smart_crop, brand_extract, segment, lut_extract
• Export — export_nle (FCPXML), translate_srt

Requirements

Requirement	Needed for	Notes
Node ≥ 18	everything
FFmpeg + ffprobe	everything	brew install ffmpeg. Invoked as a separate process; set ffmpeg_path to point elsewhere
Chrome / Chromium	render, screenshot, capture	Auto-detected; override with CLEYA_CHROME. Flags via CLEYA_CHROME_FLAGS
FFmpeg with libfreetype	card, caption, autosub	Default Homebrew formula lacks it: brew tap homebrew-ffmpeg/ffmpeg && brew install homebrew-ffmpeg/ffmpeg/ffmpeg
whisper.cpp	transcribe, autosub, clean_speech, animated_captions	Returns WHISPER_NOT_FOUND if absent
macOS + Xcode CLT	contrast, align, detect_text, caption_extract, cutout, segment, bg_remove, animated_captions, image_similarity	Vision-backed, so these nine need a Mac. Native helpers compile on first use and cache
OPENAI_API_KEY	tts only	Opt-in cloud; nothing else calls out

Cleya runs on macOS, Linux and Windows. 53 of the 67 tools are platform-independent, including the whole core loop: render, draft, starters, edit, harvest, scrub, compose, brand_check, palette, claims and diff. Nine are macOS-only (the Vision-backed row above), and five degrade rather than fail: review keeps its FFmpeg checks, smart_crop falls back to a centre crop, thumbnails loses saliency ranking, reformat loses crop:auto, autosub loses the animated style.

cleya doctor checks all of it at once.

Result contract

{ success: true, output: "/abs/path", tool, cleya_version, duration_ms, ...toolSpecificData }
{ success: false, error: "Human-readable message", code: "ERROR_CODE", suggestion?: "What to do" }

Codes are stable and actionable — VALIDATION_ERROR, FFMPEG_ERROR, WHISPER_NOT_FOUND, DRAWTEXT_NOT_FOUND, OUTPUT_EXISTS, NO_CANDIDATES, and more. Agents branch on the code; humans read the message.

Configuration

const config = resolveConfig({
  ffmpeg_path: "/opt/homebrew/bin/ffmpeg",
  temp_dir: "/tmp/cleya",
  ffmpeg_timeout: 600_000,
});

Key	Default	Purpose
ffmpeg_path / ffprobe_path	"ffmpeg" / "ffprobe"	Binary locations
whisper_path	"whisper"	whisper.cpp binary
temp_dir	"/tmp/cleya"	Working files
retention_hours	24	Temp cleanup
output_dir	"."	Default output directory
ffmpeg_timeout / whisper_timeout	300000 / 600000	Execution timeouts (ms)
platform_presets	see presets.ts	Per-platform size/codec/duration

Render tuning lives in the environment: CLEYA_RENDER_WORKERS (parallel frame capture), CLEYA_NO_GPU=1 (software raster), CLEYA_RENDER_ENGINE=chrome|webkit, CLEYA_STARTERS_DIR.

Development

npm run build        # tsc
npm test             # vitest — 834 tests across 75 files
npm run smoke        # end-to-end render smoke test

Starter scenes are contract-tested: every shipped starter must compile with zero errors and zero warnings against the neutral default brand, so a change that breaks a starter breaks the build.

License

Apache 2.0. See NOTICE for third-party attribution — in particular GSAP, which is used by the raw-HTML runtime, is licensed under GreenSock's standard "no charge" license (not Apache-2.0) and is installed from npm rather than redistributed here.