YouTube MCP Server

A Python-based Model Context Protocol (MCP) server that extracts educational content from YouTube videos — transcripts and visually meaningful frames — and makes it available to AI assistants like Claude, ChatGPT, and any MCP-compatible client.

---

What It Does

Give an AI assistant a YouTube URL, and it can:

1. Read the transcript — Full captions with timestamps, ready for summarization, Q&A, or content analysis.
2. See the video — A 5-phase computer-vision pipeline extracts the most informative frames, scores them for quality, removes duplicates, and returns them as base64-encoded JPEGs the AI can "look at."

No manual downloading. No copy-pasting. Just paste a URL and ask questions.

---

Tools

get_transcript

Fetches the video's captions using youtube-transcript-api.

Parameters:

• url (required) — YouTube video URL
• language (optional, default: "en") — Preferred caption language
• prefer_manual (optional, default: true) — Prefer manually-written captions over auto-generated

Returns: A clean, timestamped transcript block:

[00:00:00] So I want to start by offering you a free ...
[00:00:15] The key insight here is that ...

---

get_video_frames

Runs a full video-analysis pipeline to surface the most meaningful visual moments.

Parameters:

• url (required) — YouTube video URL
• max_frames (optional, default: 20) — Maximum frames to return (hard cap: 40)
• scene_threshold (optional, default: 0.25) — FFmpeg scene-detection sensitivity
• output_width (optional, default: 640) — Width of returned JPEGs (proportional height)
• min_importance_score (optional, default: 0.35) — Minimum composite quality score

Returns:

• Metadata: video_id, duration_seconds, pipeline_stats, fallback_used
• Frame blocks: each includes timestamp_ms, composite_score, and a base64-encoded JPEG image
• An index listing every returned frame with its timestamp and score

---

The Frame Pipeline (5 Phases)

Phase	Module	What It Does
1. Download	downloader.py	Downloads the video via yt-dlp with a duration guard and circuit-breaker for rate-limiting
2. Extract	frame_extractor.py	Detects scene changes with ffmpeg and extracts candidate frames
3. Score	scorer.py	Scores each frame across 5 signals: motion stability, entropy, edge density, rectangular coverage, and OCR word count
4. Deduplicate	deduplicator.py	Removes near-identical frames using DCT perceptual hashing (pHash) with a Hamming-distance threshold
5. Load	frame_loader.py	Resizes frames, compresses to JPEG (quality 85), and base64-encodes them for MCP transport

---

Quick Start

Prerequisites

• Python 3.11
• ffmpeg — video processing
• Tesseract OCR — text detection in frames

macOS:

brew install ffmpeg tesseract

Ubuntu / Linux:

sudo apt-get install ffmpeg tesseract-ocr tesseract-ocr-eng

Windows: Download ffmpeg and Tesseract, then add both to your PATH.

Verify:

ffmpeg -version
tesseract --version

Installation

# Clone the repository
git clone https://github.com/Ayush-Mamgain/youtube-mcp-server.git
cd youtube-mcp-server

# Create a virtual environment
python3 -m venv venv
source venv/bin/activate        # macOS / Linux
# venv\Scripts\activate      # Windows

# Install dependencies
pip install -r requirements.txt

Configuration

Create a .env file in the project root:

LOG_LEVEL=DEBUG
MCP_HTTP_PORT=8000

Optional tuning (loaded from config.py defaults if omitted):

Variable	Default	Description
MAX_VIDEO_DURATION	3600	Reject videos longer than this (seconds)
MAX_FRAMES_HARD_LIMIT	40	Absolute cap on returned frames
SCENE_THRESHOLD_DEFAULT	0.25	FFmpeg scene-change threshold
MIN_IMPORTANCE_SCORE_DEFAULT	0.35	Minimum composite frame score
OUTPUT_WIDTH_DEFAULT	640	Width of returned JPEGs

Start the Server

python server.py

The server starts at http://localhost:8000.

---

Connecting to Claude.ai (Local)

1. Start the server: python server.py
2. Go to Claude.ai → Settings → Integrations → Add MCP server
3. Enter: http://localhost:8000/mcp

For cloud-hosted Claude to reach your server, you'll need to expose it publicly (see Deployment below).

---

API Endpoints

Endpoint	Method	Description
GET /healthz	GET	Health check — verifies ffmpeg, tesseract, yt-dlp, and server readiness. Returns {"status": "ok"} or {"status": "error", "detail": "..."}
POST /mcp	POST	Streamable HTTP endpoint for MCP tool calls

---

Project Structure

youtube-mcp-server/
├── server.py              # MCP entry point — FastMCP + Starlette HTTP server
├── config.py              # Loads and validates environment variables
├── logger.py              # stderr-only logging
├── url_parser.py          # Validates YouTube URLs and extracts video IDs
├── transcript.py          # Fetches captions via youtube-transcript-api
├── downloader.py          # Phase 1 — video download with yt-dlp
├── frame_extractor.py     # Phase 2 — scene-change frame extraction via ffmpeg
├── scorer.py              # Phase 3 — multi-signal frame scoring (OpenCV + Tesseract)
├── deduplicator.py        # Phase 4 — perceptual-hash deduplication
├── frame_loader.py        # Phase 5 — resize, JPEG encode, base64
├── video_frames.py        # Orchestrates Phases 1–5 with semaphore and cleanup
├── requirements.txt       # Pinned Python dependencies
└── .gitignore             # Excludes .env, venv, caches, test artifacts

---

Development Workflow

This project was built in 9 self-contained stages, each with its own test file and verification step:

Stage	Focus	Test File
1	Project scaffold, config, logger	test_stage1.py
2	YouTube URL parser & validation	test_stage2.py
3	Transcript fetching	test_stage3.py
4	Video downloader with duration guard	test_stage4.py
5	Frame extraction via ffmpeg	test_stage5.py
6	Multi-signal importance scoring	test_stage6.py
7	pHash deduplication + base64 loading	test_stage7.py
8	Full pipeline orchestration	test_stage8.py
9	MCP server entry point + health checks	test_stage9.py

Golden rule: Each stage is verified before proceeding. Run python test_stage{N}.py to validate.

---

Deployment

Docker support is planned but not yet configured. This section will be updated once containerization is complete.

For now, the server runs directly on any machine with Python 3.11, ffmpeg, and Tesseract installed. Recommended hosting options once Docker is ready:

• Railway
• Fly.io
• Render

Once deployed, update your Claude.ai integration URL to:

https://YOUR-DEPLOYMENT-URL/mcp

---

Design Notes

• All logging goes to stderr only. stdout is reserved exclusively for MCP communication.
• Thread-safe: The get_video_frames pipeline uses a threading.Semaphore(1) to prevent concurrent downloads from overwhelming the system.
• Automatic cleanup: Temporary files created during video processing are deleted after each run.
• Strict validation: Video IDs are validated against ^[A-Za-z0-9_-]{11}$. Playlist-only URLs, channel URLs, and malformed inputs are rejected with clear error messages.
• Graceful degradation: If all frames score below the minimum importance threshold, the pipeline falls back to the top 5 frames and flags fallback_used: true.

---

License

MIT © Ayush Mamgain

---

Acknowledgments

Built with FastMCP, Starlette, yt-dlp, youtube-transcript-api, and OpenCV.