idml-engine

Headless InDesign document composition. Parse a designer's IDML, convert editorial content to styled ICML stories, and compose complete newspaper IDML documents — no InDesign required at build time.

Built to replace two things at a working newspaper (The Wayland Post):

• DocsFlow (the commercial Google Docs → InDesign plugin) for one-way, batch content placement, and
• a Scripts-panel ExtendScript handoff that needed a human to click a button inside InDesign for every build.

An issue that took a staffed InDesign session now composes in about one second, headlessly, and the editor opens a finished IDML to refine.

How it works

content (Google Docs / DOCX / Markdown / HTML / JSON)
   │  ingest.py — normalize to typed blocks (headline, byline, body, cutline…)
   ▼
typed blocks ──► story_converter.py ──► ICML stories   (Pandoc, style-mapped)
   │
   ▼
composer.py — "donor package" composition
   • a real published-issue IDML (or purpose-built template) donates
     Resources/ (all styles, fonts, colors) and MasterSpreads/ (the grid)
   • generates fresh Spreads/ + Stories/: standing head, headline frame,
     image grid with cutline captions, threaded body legs, bottom ad well
   ▼
complete .idml — opens in InDesign with the paper's exact design system

template_parser.py is the intelligence layer: it reads any IDML into a manifest — style catalog, master geometry with computed column ranges, every frame with threading, story-role classification (headline/body/caption/jump), and DocsFlow live-links (story → Google Doc id, decoded merge base).

Quickstart

pip install idml-engine[all]     # needs Pandoc on the machine for conversion

# understand a template or issue
idml-engine-parse issue.idml --summary

# convert a story
idml-engine-convert article.docx -o article.icml

# compose an issue (donor supplies styles/masters)
idml-engine-compose --donor published-issue.idml --manifest manifest.json --out new-issue.idml

Python API:

from idml_engine import composer, ingest, story_converter, template_parser

manifest = template_parser.parse_template("issue.idml")
blocks   = ingest.from_file("article.docx")
story_converter.blocks_to_icml(blocks, "article.icml")
composer.compose("donor.idml", articles, "out.idml", ads=ads)

Layout priors — "trained" on your own paper

The composer doesn't guess layout metrics; it learns them from your archive. layout_learner.py mines a corpus of published-issue IDMLs into layout-priors.json — measured body-text density (chars/pt², drives column capacity), headline-frame heights, caption heights, image sizes, ad-band heights, and stories-per-page (quartile distributions). The composer auto-loads priors (env IDML_ENGINE_PRIORS, or the file next to the package) and falls back to model constants without them.

python -m idml_engine.layout_learner "path\to\published issues" -o layout-priors.json

At The Wayland Post: 27 issues mined → 729 body-story density samples, 937 headline frames, 1,200 placed images. The measured density (~31 chars/line) corrected the hand-tuned model's 40 by 25%.

MCP server (AI-drivable)

Every capability is exposed as MCP tools, so Claude (or any MCP client) can parse templates and compose issues conversationally:

claude mcp add idml-engine -- python -m idml_engine.mcp_server

Tools: parse_template, list_template_styles, get_template_geometry, validate_template, ingest_content, convert_story, compose_issue, compose_articles.

DocsBridge panel — the DocsFlow-style UI

uxp-panel/ is an InDesign UXP panel that replicates DocsFlow's workflow on this engine: load an issue → per-article Place (drops the styled ICML story and live-links it via storyTitle="docx:<google-doc-id>" — DocsFlow's own convention) → Update re-places into the linked story's first frame, preserving the frame chain. Status badges (placed / not placed / no copy) survive save/reopen because the link rides in the document.

Backed by idml_engine/panel_service.py (localhost:3100 — manifest listing, on-demand blocks→ICML, Drive refetch). Load the panel once via the Adobe UXP Developer Tool (uxp-panel/manifest.json).

Commercial REST API

commercial/api_server.py wraps the same engine with Bearer-key auth and per-key usage metering for hosted deployment (FastAPI):

uvicorn commercial.api_server:app --port 8080
# POST /api/compose  /api/parse  /api/convert   (Authorization: Bearer <key>)
# GET  /api/usage

Keys + metering are file-backed stubs; production wires them to Stripe metered billing (see commercial/README.md).

Findings that made this possible

• ICML style mapping: Pandoc's ICML writer (stable since 2014) plus a post-pass that strips its " > Paragraph" style-name suffix yields stories whose styles match the template by exact name — InDesign adopts the template's formatting on Place. Trailing-space style names ("Byline ") survive round-trip.
• Donor-package composition: a finished issue's IDML already carries the paper's whole design system. Cloning its resources and generating only spreads + stories produces documents InDesign opens cleanly.
• DocsFlow link format: DocsFlow stores its Google Doc live-links in plain sight — StoryTitle="docx:<file-id>" plus a merge-base blob that is just base64 → raw DEFLATE → JSON. A future release can read, preserve, and write these links for drop-in compatibility.

See docs/TEMPLATE-STRUCTURE.md for the full IDML anatomy notes.

Status

Alpha. Working end-to-end at one real newspaper. Not yet implemented: multi-article pages, jump lines ("Continued on…"), text wrap around images, DocsFlow-compatible link writing, IDML→PDF export (open in InDesign, or pair with InDesign Server / the UXP MCP bridge).

License

BSD-3-Clause (this library). The commercial API layer and hosted service are separate products built on top.