Media2HTML: A VLM-Free Pure Compiler for Multimodal Agents

Disclaimer: Please understand this is V0.01 and still very experimental. It is the product of a personal theory and nothing more.

A tool that converts images, videos, and audio into structured, cacheable HTML representations optimized for text-based LLM reasoning.

V0.0.1 — Pure Compiler Architecture

---

Concept

Modern multimodal AI agents need to reason about media files, but LLMs are fundamentally text-based. Traditional approaches — passing raw base64 images or unstructured VLM captions — suffer from massive token overhead, hallucination risk, and GPU contention.

Media2HTML eliminates all of this. It extracts structured facts (objects, text, colors, spatial relations, semantic tags) using deterministic algorithms, then compiles them into semantic HTML that LLMs understand natively. Zero Vision-Language Models in the runtime loop.

Key benefits:

• (To be dertimined) token reduction vs. raw base64 images
• 0 GB VRAM — all extraction runs on CPU, leaving GPU free for the text LLM
• 100% deterministic — same input always produces the same output
• Zero hallucination — no generative model in the pipeline
• Instant caching — repeated file analysis returns in <1ms

---

V0.0.1 Stack

Component	Technology	Purpose
Object Detection	YOLO-World + SAHI	Open-vocabulary detection with sliced inference for 4K+ images
Z-Axis Layout	Depth-Anything V2	Foreground / midground / background categorization
Semantic Tags	CLIP (openai/clip-vit-base-patch32)	Zero-shot scene tagging (rain, night, cherry blossom, etc.)
Text Extraction	RapidOCR	CPU-only OCR with paragraph grouping
Visual Maps	ASCILINE	Token-optimized 512-cell grid for visual reasoning
Audio	Whisper + Pyannote	Transcription and speaker diarization
Video	scenedetect + FFmpeg	Scene detection with keyframe extraction

---

Setup

1. Install System Dependencies

sudo apt install ffmpeg portaudio19-dev

2. Install Python Packages

pip install -e .

3. Environment Variables (Optional)

export HF_TOKEN="hf_..."  # Required for Pyannote audio diarization

---

Usage

As a Python Library

from media2html import media_to_html

# Image
html = media_to_html("path/to/image.jpg", mode="rich")

# Video (with interleaved audio)
html = media_to_html("path/to/video.mp4", mode="compact")

# Audio
html = media_to_html("path/to/audio.wav", mode="minimal")

As an MCP Server

# Start the MCP server
python3 -m media2html.mcp_server

# Or run directly
python3 media2html/mcp_server.py

Agents call the transcode_media tool to analyze any local media file.

---

Modes

Mode	Content	Token Count
minimal	Caption + top objects + colors	~200
compact	+ OCR + visual grid	~800
rich	+ spatial relations, semantic tags, depth scene, accent colors	~2500

---

Output Format

Media2HTML produces semantic HTML that leverages LLM pretraining priors:

<image-summary width="1376" height="768" source="photo.png">
  <caption>Scene contains: car, person, umbrella. Text detected: 'LOFUGA'. Dominant color: #303048. Tags: rain, wet road.</caption>
  <spatial-graph>
    <rel>car is left of person</rel>
    <rel>person is above car</rel>
  </spatial-graph>
  <semantic-tags>
    <t>rain</t>
    <t>wet road</t>
  </semantic-tags>
  <scene>
    <foreground>car</foreground>
    <background>person, umbrella</background>
  </scene>
  <accent-colors>
    <c hex="#e0c4c4" pct="0.04" label="vibrant"/>
  </accent-colors>
  <objects>
    <obj label="car" bbox="0.277,0.438,0.758,1.000"/>
  </objects>
  <text-regions>
    <t bbox="0.350,0.750,0.412,0.794">LOFUGA</t>
  </text-regions>
  <colors>
    <c hex="#303048" pct="0.08"/>
  </colors>
</image-summary>

---

Architecture

media2html/
├── pyproject.toml           # Package configuration
├── requirements.txt         # Dependencies
├── README.md                # This file
├── TECHNICAL_WHITEPAPER.md  # Full architecture documentation
├── test_media2html.py       # Test suite
└── media2html/
    ├── __init__.py          # Package exports
    ├── cache.py             # Disk-based caching (diskcache + MD5)
    ├── html_builder.py      # Data model + HTML generation
    ├── pipeline.py          # Main extraction pipeline
    ├── mcp_server.py        # MCP server for agent integration
    └── extractors/
        ├── __init__.py
        ├── vision.py        # YOLO-World + Depth-Anything + CLIP + RapidOCR
        ├── audio.py         # Whisper + Pyannote
        └── video.py         # Scene detection + keyframes

---

Performance

Metric	Value
Token reduction vs. raw base64	to be determined
Extraction latency (minimal)	~200ms
Extraction latency (rich)	~700ms
GPU VRAM required	0 GB (all CPU)
Deterministic output	100%
Hallucination risk	0%
Cache hit latency	<1ms

---

License Creative Commons