VAAS

<p align="center"> <img src="assets/vaas-hero.svg" alt="VAAS — Vision as a Service" width="100%" /> </p>

<p align="center"> <strong>A visual memory and attention layer for AI agents.</strong><br/> Index images and long videos once. Search, inspect, track, and retrieve the right frame in milliseconds. </p>

<p align="center"> <a href="#60-second-demo">60-second demo</a> · <a href="#mcp-server">MCP server</a> · <a href="#how-it-fits-together">Architecture</a> · <a href="#model--library-credits">Credits</a> </p>

---

VAAS is a compact prototype for treating vision as a queryable data source, not a stream of disposable screenshots. It gives an agent a CLI and an MCP server for image portfolios, recorded video, sampled camera feeds, attention timelines, coarse face signals, and persistent visual entities.

This is portfolio-grade infrastructure: real, testable, and deliberately modular. It is not a production biometric, surveillance, or safety-critical system.

See VAAS in action

<p align="center"> <a href="assets/vaas-demo.mp4"> <img src="assets/vaas-demo.gif" alt="VAAS product demonstration: visual indexing, search, attention tracking, and MCP tools" width="800" /> </a> </p>

<p align="center">_{19-second overview · click the animation for the full-quality MP4}</p>

What it demonstrates

Need	VAAS primitive	Included implementation
Search a large visual collection	image/text embeddings + cosine retrieval	tiny local descriptor; optional OpenCLIP
Find moments in long video	sampled frame index + scene/motion scores	OpenCV adapter
Keep attention over time	normalized saliency grid + focus centroid	NumPy contrast/edge/saturation pipeline
Recognize reactions	face boxes + smile signal	optional OpenCV Haar adapter
Resolve repeated subjects	online prototype centroids + observations	SQLite entity registry
Let any agent use it	typed tools over stdio	official MCP Python SDK
Pull the evidence	frame export with source/time provenance	image copy or precise video seek

60-second demo

Requires Python 3.10+.

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

# Creates a tiny synthetic portfolio, indexes it, and runs visual search.
python examples/quickstart.py

Or point the CLI at real media:

vaas index ~/Pictures/portfolio --tag portfolio
vaas search portfolio --limit 5
vaas search --image ~/Pictures/query.jpg --limit 5
vaas inspect asset_abc123

Video and coarse facial signals are opt-in:

pip install -e '.[video]'
vaas --face-signals index demo.mp4 --sample-every 1.5 --tag interview
vaas timeline demo.mp4
vaas export asset_abc123 exports/interesting-frame.jpg

The default descriptor is intentionally tiny and supports example-image similarity. For semantic text-to-image search, select OpenCLIP:

pip install -e '.[semantic]'
vaas --embedder openclip index ~/Pictures/portfolio
vaas --embedder openclip search "a red bicycle beside a brick wall"

The first OpenCLIP run downloads the selected model weights. A catalog only compares vectors produced by the same backend, so index and search with the same --embedder value.

MCP server

Install the official SDK adapter and run over stdio:

pip install -e '.[mcp,video]'
vaas --db ~/.local/share/vaas/catalog.db serve

Example MCP client configuration:

{
  "mcpServers": {
    "vaas": {
      "command": "/absolute/path/to/.venv/bin/vaas",
      "args": ["--db", "/absolute/path/to/vaas.db", "serve"]
    }
  }
}

The agent receives seven focused tools:

MCP tool	Agent use
visual_status	Check the catalog, database, and active embedder
index_visual_path	Build memory from one file, a directory, or a video
search_visual_memory	Search by text/tags or an example image
inspect_visual_asset	Read provenance, attention, signals, and metadata
read_attention_timeline	Follow focus, motion, and shot changes over time
resolve_visual_entity	Map an observation to a stable visual subject
export_visual_frame	Materialize the source frame for multimodal inspection

A useful agent loop looks like this:

index_visual_path("meeting.mp4", sample_every_seconds=1)
→ search_visual_memory("whiteboard", limit=3)
→ inspect_visual_asset(best_asset_id)
→ export_visual_frame(best_asset_id, "exports/whiteboard.jpg")
→ agent inspects the exported image with its native vision model

Python API

from vaas import VAAS

vision = VAAS("vaas.db", embedder="visual", face_signals=False)
frames = vision.index_video("meeting.mp4", sample_every=2.0, max_frames=300)

matches = vision.search(image="whiteboard-photo.jpg", limit=5)
best = matches[0]
print(best.record.source_uri, best.record.timestamp, best.record.attention)

entity = vision.resolve_entity(best.record.id, kind="scene", label="planning board")
vision.export_frame(best.record.id, "exports/planning-board.jpg")

See examples/agent_workflow.py for a complete media-to-evidence flow.

How it fits together

flowchart LR
  A["Images · video · camera snapshots"] --> B["Sampler + provenance"]
  B --> C["Pluggable classifiers"]
  C --> D["Visual embeddings"]
  C --> E["Attention + face signals"]
  D --> F[("SQLite visual catalog")]
  E --> F
  F --> G["Similarity + metadata search"]
  F --> H["Entity prototypes"]
  G --> I["CLI"]
  H --> I
  G --> J["MCP tools"]
  H --> J
  J --> K["Any multimodal agent"]
  K --> L["Export exact frame"]

The package has one orchestration API, VAAS, used by both interfaces. Adapters conform to a small Embedder protocol. SQLite holds metadata and float32 vectors; NumPy performs exact cosine search. That makes the demo transparent and portable. At portfolio scale, swap the vector scan for Faiss, Qdrant, Milvus, or pgvector without changing the agent tools.

Data model

source ──< asset/frame ──> embedding
                 │
                 ├── attention {score, focus_x, focus_y, entropy, 8×8 grid}
                 ├── signals   {faces, smiles, adapter-specific outputs}
                 └── observation >── entity {kind, label, running centroid}

Every result preserves its original source, video timestamp, frame number, content hash, dimensions, embedding model, and analysis metadata. VAAS stores indexes—not copied media—unless export is explicitly called.

Extending the sensing library

VAAS does not vendor model code. Add an adapter and keep weights/licenses with their upstream project:

class MyEmbedder:
    name = "my-model:v1"

    def embed_image(self, image):
        return normalized_numpy_vector

    def embed_text(self, text):
        return normalized_numpy_vector


vision = VAAS("catalog.db", embedder=MyEmbedder())

Good next adapters include:

• DINOv2 for general-purpose visual similarity and entity features.
• SAM 2 for promptable object masks and temporal object tracking.
• MediaPipe Face Landmarker for blendshapes, head pose, and richer interaction signals.
• PySceneDetect for production-grade shot boundaries.
• Faiss/HNSW for million-scale approximate nearest-neighbor search.

Boundaries and responsible use

• The built-in face adapter detects coarse face/smile patterns. It does not identify people, infer emotion, or establish intent.
• Smile detection is a noisy visual signal, not an emotional truth. Treat all facial outputs as uncertain observations.
• Obtain consent before processing cameras, calls, faces, or private media. Follow retention and access-control requirements.
• Keep a human in the loop for consequential uses. Benchmark every selected model on the actual domain and demographic mix.
• Validate paths and add authentication/authorization before exposing the MCP server beyond a trusted local process.

Model & library credits

VAAS uses or is designed to interoperate with these excellent projects. Their code is referenced through normal dependencies; none is copied into this repository.

• Pillow and NumPy — image I/O and local numeric features.
• SQLite — portable catalog and entity registry.
• OpenCV — optional video decoding, shot/motion statistics, Haar face/smile signals.
• OpenCLIP — optional text/image embeddings; based on Radford et al., Learning Transferable Visual Models From Natural Language Supervision.
• MCP Python SDK — the stdio agent interface.
• Oquab et al., DINOv2: Learning Robust Visual Features without Supervision — recommended general visual features adapter.
• Ravi et al., SAM 2: Segment Anything in Images and Videos — recommended mask and object-tracking adapter.
• Lugaresi et al., MediaPipe: A Framework for Building Perception Pipelines — recommended real-time landmark/blendshape adapter.
• Johnson, Douze, and Jégou, Billion-scale similarity search with GPUs and Faiss — recommended large-scale vector index.

Development

pip install -e '.[dev]'
pytest -q
ruff check .

# Regenerate the README MP4 and GIF (requires ffmpeg).
python scripts/render_readme_demo.py

The MIT license covers VAAS itself. Optional models, weights, and dependencies retain their own licenses and usage terms.