SanTech Content Engine

Local-first YouTube research-to-script engine for SanTech010.

The backend performs deterministic work: YouTube fetching, storage, outlier math, pack building, validation, save/select/export, and MCP tool exposure. Connected AI clients perform creative and judgment work using backend packs as source of truth.

Current status: offline backend, MCP backbone, quota guard, resource import, recent-only outliers, early velocity signals, combined signal reports, and signal-target transcript acquisition implemented.

Important MCP rule: source files sync locally on MCP HTTP startup and before YouTube collection. get_signal_research_pack reads the current local database. It does not pull fresh YouTube data. Use get_project_status to check freshness and source-sync warnings.

Verified:

python -m pytest tests -q
python -m compileall -q src

New Laptop Setup

On a fresh Windows laptop:

git clone git@github.com:budhasantosh010/san-tech-yt-research-titles-scripts.git
Set-Location -LiteralPath ".\san-tech-yt-research-titles-scripts"
powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\bootstrap-windows.ps1

Then edit .env locally and add:

• YOUTUBE_API_KEYS
• MCP_ROUTE_SECRET
• ALLOWED_HOSTS with your Tailscale Funnel hostname

Do not commit .env.

Before pushing from any laptop:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\check-github-ready.ps1

Full reproducibility instructions live in DOCS/FUTURE_PROOF_SETUP.md.

Daily MCP Startup

Use two PowerShell terminals.

Terminal 1:

$project = "C:\Users\Lenovo\Music\Startups\YT Automations\1 San Tech Channel Research, Title & Script with Codex"
Set-Location -LiteralPath $project
.\scripts\run-mcp-http.ps1

Terminal 2:

$project = "C:\Users\Lenovo\Music\Startups\YT Automations\1 San Tech Channel Research, Title & Script with Codex"
Set-Location -LiteralPath $project
.\scripts\start-funnel.ps1

Good Funnel output should show:

proxy http://127.0.0.1:3001

ChatGPT Connector URL Format

Use this shape:

https://<machine>.<tailnet>.ts.net/mcp/YOUR_ROUTE_SECRET/

Never include brackets around route secret.

Correct:

https://desktop-fdce9ak.taila47816.ts.net/mcp/abc123.../

Wrong:

https://desktop-fdce9ak.taila47816.ts.net/mcp/<abc123...>/

Use one tool call at a time in ChatGPT. No parallel tool calls. For connection tests, call only get_project_status.

Implemented:

• SQLite schema
• config loader
• competitor storage
• video/snapshot storage
• recent-only outlier math and persistence
• early velocity signal math and persistence
• YouTube Data API adapter with fake-client tests
• viral title collection importer for CSV/XLSX
• video idea save/list/select
• title idea save/list/select
• script points save/select
• context pack builders
• transcript save/get/list-missing storage
• signal-target transcript acquisition adapters
• transcript acquisition attempt tracking
• transcript-aware outlier research packs
• script save/export
• FastMCP server wrapper
• workflow prompt templates

Still needs live/user-provided inputs:

• YOUTUBE_API_KEY
• YOUTUBE_API_KEYS for comma-separated multiple keys
• competitor channel list
• real viral title/video collection file
• ChatGPT MCP connector/tunnel setup

Quota safety:

• YOUTUBE_MAX_DAILY_QUOTA_FRACTION=0.05 limits the app to 5% of each key's configured daily quota.
• YOUTUBE_CACHE_MAX_AGE_HOURS=24 prevents refetching fresh video details/snapshots.
• RESEARCH_STALE_AFTER_HOURS=24 controls when get_project_status warns that research data is stale.
• AUTO_SYNC_SOURCES_ON_STARTUP=true keeps competitor and viral sources synced locally when the MCP HTTP server starts.
• AUTO_SYNC_DEACTIVATE_MISSING=false means startup sync repairs/reactivates source rows but does not deactivate missing competitors unless explicitly enabled.
• API keys belong in .env, which is ignored by Git.

Resource files:

• Competitors source of truth: resources/competitors/santech_competitors.txt
• Viral titles: resources/viral titles in adjacent niches

Competitor TXT format:

• one absolute YouTube channel URL per line
• no display names
• no notes
• no commas
• no copied YouTube search/export rows

Messy CSV/XLSX/MD/TXT files can be inspected/imported, but do not use them as the canonical competitor source. If a sync source has zero files, zero rows, or zero valid YouTube channel URLs, deactivation is skipped for safety.

Signal workflow

Current deterministic workflow:

1. Update resources/competitors/santech_competitors.txt if competitor sources change.
2. Update viral title CSV/XLSX files if the viral title collection changes.
3. Start the MCP HTTP server; backend auto-syncs sources locally.
4. Refresh YouTube data and save a new snapshot row per collection run.
5. Calculate recent outliers.
6. Calculate early velocity.
7. Dump a combined signal report.
8. Fetch transcripts for selected signal targets only.
9. Build transcript-aware research packs.
10. Later, ChatGPT analyzes and generates ideas/titles/scripts.

MCP tool boundaries:

• Source sync is automatic on backend startup and before collect_all_competitors.
• sync_competitors_from_file and sync_viral_collection remain admin/debug tools, but ChatGPT should not call them during normal use.
• scripts/sync-sources.ps1 runs source sync locally without ChatGPT.
• scripts/run-research-refresh.ps1 runs local source sync, YouTube collection, signal calculation, and project status from PowerShell.
• refresh_research_data can collect YouTube stats and recalculate outlier/velocity signals. It does not fetch transcripts.
• get_signal_research_pack is the read tool ChatGPT should use after freshness is acceptable.
• mark_signal_reviewed can mark a signal as reviewed, approved, rejected, used, or suppressed; default reports exclude reviewed/rejected/used/suppressed signals.

Recent outlier means a recent video already beat that channel's recent baseline.

Early velocity means a new video is moving unusually fast and may become an outlier.

True recent velocity requires at least two snapshots. If only one snapshot exists, the system uses lifetime velocity as a rough fallback.

Canonical competitor source sync:

python -m santech_engine.cli sync-sources

Signal commands:

python -m santech_engine.cli collect-all --limit 10
python -m santech_engine.cli calculate-outliers --recent-only --all-competitors
python -m santech_engine.cli list-outliers --limit 5
python -m santech_engine.cli dump-outlier-report --limit 20 --output exports/recent-outliers.json
python -m santech_engine.cli calculate-velocity --all-competitors
python -m santech_engine.cli list-velocity-signals --limit 10
python -m santech_engine.cli dump-velocity-report --limit 20 --output exports/early-velocity.json
python -m santech_engine.cli dump-signal-report --output exports/signal-report.json
python -m santech_engine.cli fetch-transcripts-for-signal-targets --adapter youtube_transcript_ai --limit-outliers 5 --limit-velocity 5

Transcript acquisition

This project supports transcript-aware research packs.

The system does not fetch transcripts for every competitor video.

Flow:

1. Collect competitor videos.
2. Calculate recent outliers and early velocity.
3. Fetch transcripts only for combined signal targets.
4. Cache successful transcripts.
5. Mark missing/failed transcripts honestly.
6. Include real transcript text in outlier packs only when available.

Default adapter:

• youtube_transcript_ai

Backup adapter:

• youtube_transcript_io if YOUTUBE_TRANSCRIPT_IO_API_TOKEN is set

Manual fallback:

• If automatic fetching fails for an important outlier, the user can paste transcript text into ChatGPT.
• ChatGPT can then call save_video_transcript through MCP to cache it.

Warnings:

• Transcript providers are not the official YouTube Data API.
• They may fail or change.
• Do not use aggressive scraping/proxy bypassing.
• Keep fetches low-volume and signal-first.
• The system must not infer or invent what a video said from title/metrics alone.

Useful local commands:

python -m santech_engine.cli import-competitors
python -m santech_engine.cli import-viral-collection
python -m santech_engine.cli doctor
python -m santech_engine.cli setup-env
python -m santech_engine.cli test-transcript-provider --adapter youtube_transcript_ai
python -m santech_engine.cli smoke-test
python -m santech_engine.cli collect-all --limit 5
python -m santech_engine.cli calculate-outliers --recent-only --all-competitors
python -m santech_engine.cli calculate-velocity --all-competitors
python -m santech_engine.cli dump-signal-report --output .\exports\signal-report.json
python -m santech_engine.cli fetch-transcripts-for-signal-targets --adapter youtube_transcript_ai --limit-outliers 5 --limit-velocity 5
python -m santech_engine.cli transcript-status --video-id 123
python -m santech_engine.cli list-transcript-attempts --limit 20
python -m santech_engine.cli save-transcript --video-id 123 --file .\transcript.txt
python -m santech_engine.cli dump-outlier-pack --include-transcripts --output .\exports\outlier-pack.json
python -m santech_engine.cli dump-title-pack --idea-id 1 --output .\exports\title-pack.json
python -m santech_engine.cli dump-script-pack --title-id 1 --output .\exports\script-pack.json

Run MCP server locally:

python -m santech_engine.mcp_server

ChatGPT MCP via Tailscale Funnel

Use this only after the local backend packs are truthful.

1. Create local .env:

Copy-Item .env.example .env

2. Set the MCP bridge values in .env.

MCP_HOST=127.0.0.1
MCP_PORT=3001
MCP_ROUTE_SECRET=<your-long-secret>
MCP_BEARER_TOKEN=
ALLOWED_ORIGINS=https://chatgpt.com,https://chat.openai.com
ALLOWED_HOSTS=127.0.0.1,localhost,<your-machine>.<your-tailnet>.ts.net
FUNNEL_BASE_URL=

Rules:

• At least 32 characters.
• Do not commit .env.
• Do not paste the secret into docs.
• Keep MCP_BEARER_TOKEN blank if the ChatGPT connector uses Authentication: No Auth.
• Add your real Tailscale Funnel hostname to ALLOWED_HOSTS.

3. Run the HTTP bridge doctor:

.\scripts\doctor-mcp-http.ps1

4. Start the local MCP HTTP server:

.\scripts\run-mcp-http.ps1

5. In another terminal, start Tailscale Funnel:

.\scripts\start-funnel.ps1

6. Add this URL as the ChatGPT custom connector URL:

https://<machine>.<tailnet>.ts.net/mcp/<MCP_ROUTE_SECRET>

Use your real Tailscale Funnel hostname and your local route secret value.

7. In ChatGPT, test:

get_project_status

8. Then test the main research pack:

get_signal_research_pack

Expected route:

ChatGPT custom connector
-> Tailscale Funnel HTTPS URL
-> http://127.0.0.1:3001/mcp/<MCP_ROUTE_SECRET>
-> SanTech MCP tools
-> SQLite / YouTube API / transcript providers

Security notes:

• The Python server binds only to 127.0.0.1.
• The default SanTech MCP HTTP port is 3001.
• The MCP route is mounted under /mcp/<MCP_ROUTE_SECRET>.
• Allowed origins default to https://chatgpt.com and https://chat.openai.com.
• Allowed hosts default to 127.0.0.1 and localhost; add the Tailscale Funnel hostname through ALLOWED_HOSTS or FUNNEL_BASE_URL.
• If another MCP bridge, such as the X/Spectre bridge, is using the same Tailscale Funnel root hostname, run only one of them through that public root at a time unless you add a gateway/router.
• Scripts and doctor checks do not print API keys, bearer tokens, or the MCP route secret.
• Stop Funnel with:

.\scripts\stop-funnel.ps1