reelminner

<p align="center"> <img src="assets/icon.ico" alt="Reelminner" width="96" height="96" /> </p>

<h1 align="center">🎬 Reelminner</h1> <p align="center"><b>Instagram Reel & Profile Scraper — Pro</b></p>

<p align="center"> <img alt="Python" src="https://img.shields.io/badge/Python-3.10%2B-3776AB?logo=python&logoColor=white" /> <img alt="Playwright" src="https://img.shields.io/badge/Engine-Playwright-2EAD33?logo=playwright&logoColor=white" /> <img alt="License" src="https://img.shields.io/badge/License-MIT-green" /> <img alt="Platform" src="https://img.shields.io/badge/Platform-Windows%20%7C%20WSL%2FLinux-lightgrey" /> </p>

<p align="center"> Extract <b>reel metadata</b>, <b>owner profiles</b>, <b>follower counts</b>, <b>music</b>, <b>likes / comments / views</b> and more from any public Instagram Reel — through a desktop GUI, a one‑command CLI, a Python API, or an <b>MCP server</b> that AI agents can drive. </p>

💡 Name note: This project's final public name is Reelminner. The Python engine class is Reelminner (see scraper.py), the CLI/GUI and MCP server are branded reelminner, and the GitHub repository is reelminner. The earlier working codename ReelSnipe has been fully retired. Other name ideas are listed in Name options.

---

📚 Table of Contents

• What is Reelminner
• ✨ Features
• 🧠 How it works
• 🏗️ Project architecture
• 📦 Installation
• 🚀 Quick Start
• 💻 Usage
  • Desktop GUI
  • Command Line (CLI)
  • MCP Server (for AI agents)
  • Python API
• 📊 Output format
• ⚙️ Configuration
• 🗂️ Project structure
• 🧪 Testing & QA
• 📦 Building a standalone EXE
• ⚠️ Legal & ethical disclaimer
• 🤝 Contributing
• 📄 License
• 🆘 Troubleshooting & FAQ
• 🏷️ Name

---

What is Reelminner

Reelminner is an open‑source toolkit that pulls structured data out of Instagram Reels and the profiles that posted them. It is built around a single, reusable engine (Reelminner) that is exposed four different ways:

Interface	File	Best for
🖥️ Desktop GUI	gui.py	Non‑technical users, one‑click scraping
⌨️ CLI	scraper.py	Power users, batch jobs, scripts
🤖 MCP server	mcp_server.py	AI agents / LLM workflows
🐍 Python API	import scraper	Embedding inside your own code

Everything shares the same parsing, session, and rate‑limit logic, so results are identical no matter which front‑end you use.

---

✨ Features

• Multi‑source reel parsing — Reelminner reads data from several layers (embedded JSON, GraphQL responses, and a live DOM fallback) so it keeps working even when Instagram changes one of them.
• Owner profile enrichment — for every reel it can auto‑fetch the poster's username, full_name, bio, followers, is_verified, and reels_count.
• Follower count extraction — pulled via Instagram's GraphQL UserByRestrictedView / GraphQLOwnerInfo query, with a DOM fallback and pagination (handles capped follower figures like “1.2M” by scrolling the profile).
• Music metadata — reel audio music_title, music_artist, and music_id.
• Engagement metrics — views, likes, comments, and the direct video_url / thumbnail.
• Session & login management — interactive QR/login, cookie import from EditThisCookie exports, and a 24‑hour session refresh so you don't re‑login constantly.
• Concurrent scraping — a thread pool (--workers, default 3) with polite inter‑request delays (--delay, default 2s) and adaptive back‑off when Instagram throws BLOCKED / RATE_LIMITED.
• Resilient status tracking — every row carries a status code (OK, PARSED_PARTIAL, FAILED, NO_DATA, BLOCKED, RATE_LIMITED) so you know exactly what succeeded.
• Multiple export formats — CSV (default), JSON, and Excel (.xlsx via openpyxl).
• MCP server — five stable tools so an AI agent (Claude, Cursor, etc.) can scrape, check status, import cookies, stop, and export.
• Desktop GUI — built‑in dark theme, paste‑URL box, live results table, right‑click copy URL / open reel, and one‑click export.
• Tested — pytest suite + an end‑to‑end QA harness that enforces data‑quality gates.

---

🧠 How it works

┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐
│   GUI      │   │    CLI     │   │  MCP srv   │   │  Python    │
│  gui.py    │   │ scraper.py │   │mcp_server  │   │   import   │
└─────┬──────┘   └─────┬──────┘   └─────┬──────┘   └─────┬──────┘
      └────────────────┴────────────────┴────────────────┘
                       ▼
              ┌───────────────────────┐
              │  Reelminner  │  ← the engine (scraper.py)
              │  • session / cookies   │
              │  • thread pool         │
              │  • adaptive back‑off   │
              └───────────┬───────────┘
                          ▼
              ┌───────────────────────┐
              │  parsers.py            │  ← pure extraction helpers
              │  parse_reel_page / json│
              │  parse_owner / music   │
              │  regex adapters         │
              └───────────────────────┘

1. Normalize the input URL (normalize_reel_url) so /reel/X/ and /reel/s/…/ both work.
2. Load session — apply saved cookies (sessionid, csrftoken, ds_user_id, ig_did, mid, rur) or log in.
3. Fetch & parse the reel page with a layered fallback:
   • parse_reel_page → embedded window.__additionalData / sharedData HTML JSON
   • parse_reel_json → raw GraphQL GQL response
   • parse_graphql_reel → shortcodeMedia object
   • DOM fallback → _extract_text_raw queries the live page for likes / comments / plays / followers via regex adapters.
4. Enrich owner (unless --no-profiles): fetch the profile and read followers, full_name, bio, is_verified, reels_count.
5. Respect limits: sleep delay between requests; if blocked, back off and retry.
6. Write rows to CSV / JSON / Excel with a status per row.

---

🏗️ Project architecture

Reelminner is a single‑engine, multi‑interface design. One core engine (Reelminner) does all the real work; the GUI, CLI, MCP server, and Python API are thin front‑ends that call into it. This keeps parsing, session handling, and rate‑limiting identical across every entry point.

                         ┌─────────────────────────────┐
        URL(s) in ──────▶│     Reelminner     │  scraper.py
                         │  ── engine / orchestrator ──  │
                         └───────┬───────────┬──────────┘
                  run scrapes    │           │  enrich owner
                                 ▼           ▼
                    ┌────────────────┐  ┌──────────────────┐
                    │   parsers.py    │  │ session + graphql│
                    │ pure extractors │  │ (followers/music)│
                    └───────┬────────┘  └─────────┬────────┘
                            └─────────┬────────────┘
                                      ▼
                            ReelData row + status
                                      ▼
                       CSV / JSON / Excel writers

Module responsibilities

File	Role	Key public symbols
scraper.py	Core engine + CLI. Owns the browser, session, thread pool, and writers.	Reelminner, scrape(), login(), has_session(), save_cookies_from_file(), clear_session(), write_csv, export_json, export_excel, normalize_reel_url, csv_columns, ReelData, DEFAULT_STATE_FILE
parsers.py	Pure extraction helpers — no browser, easy to unit‑test.	parse_reel_page, parse_reel_json, parse_graphql_reel, parse_owner_username_from_html, parse_music, parse_count, parse_caption, parse_graphql_followers, parse_profile_card
gui.py	Tkinter desktop app. Builds the window, menu, URL box, workers slider, results table, and export dialogs.	ReelminnerGUI, build(), scrape(), export_*, copy_url(), open_reel()
theme.py	GUI styling — applies the dark theme to ttk widgets.	apply_dark_theme(root)
mcp_server.py	MCP server — exposes the engine as 5 tools for AI agents over stdio.	mcp (FastMCP), scrape_reels, get_status, import_cookies, stop_scrape, export_results
build_exe.py	Packaging — PyInstaller one‑file build.	EXE(...), COLLECT/Analysis
run_qa.py	QA harness — runs the engine over a corpus and enforces data‑quality gates.	run_qa(), gate checks, qa_report.json

Engine internals (Reelminner)

• Session layer — _SESSION_COOKIE_NAMES (sessionid, csrftoken, ds_user_id, ig_did, mid, rur); _apply_cookies(), _refresh_if_needed() (24h), login() (interactive QR), clear_session().
• Concurrency — scrape() spins up a ThreadPoolExecutor(max_workers=workers); each URL is handled by _worker → _scrape_url, which calls _gather_metadata (reel data) and optionally _gather_article (owner profile). A semaphore + _sleep() enforce politeness; status_code / retcode drive an adaptive retry/back‑off loop when Instagram returns BLOCKED / RATE_LIMITED.
• Parsing pipeline (layered fallback) — inside _gather_metadata the engine tries, in order: parse_reel_page (embedded HTML JSON) → parse_reel_json (raw GraphQL GQL) → parse_graphql_reel (shortcodeMedia) → DOM fallback via the _extract_text_html / _extract_text_raw adapters and the _PATTERNS regex list (likes/comments/plays/followers).
• Profile enrichment — get_follower_count() uses Instagram's GraphQL UserByRestrictedView / GraphQLOwnerInfo query, falling back to the DOM and paginating followers (_fetch_followers with end_cursor) when counts are capped.
• Output — rows are collected as ReelData dicts and written by write_csv (respecting csv_columns), export_json, or export_excel (needs openpyxl).

Why this layout

• Testability — all parsing lives in parsers.py with no browser dependency, so tests/test_parsers.py can assert on saved HTML/JSON fixtures.
• One source of truth — every interface shares the same Reelminner, so a fix in the engine benefits the GUI, CLI, and MCP server simultaneously.
• Safe packaging — the GUI/CLI thin shells mean the PyInstaller EXE only bundles the engine + a minimal UI, keeping the binary small.

---

📦 Installation

Requirements: Python 3.10+ and the Playwright browser engine.

# 1. Clone
git clone https://github.com/ilovekushgola/reelminner.git
cd reelminner

# 2. (Recommended) create a virtual environment
python -m venv .venv
.venv\Scripts\activate        # Windows
# source .venv/bin/activate   # macOS / Linux

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install the Chromium browser for Playwright
playwright install chromium

GUI only: the desktop app uses tkinter, which ships with standard Python installs. No extra package needed. The GUI is most polished on Windows.

Optional dev/test tools:

pip install -r requirements-dev.txt   # pytest, coverage

💡 Before you start: Reelminner works best with a logged‑in Instagram session — some reels and all owner/follower data require authentication. Run python scraper.py --login once (interactive QR), or import cookies exported from the EditThisCookie browser extension with python scraper.py --import-cookies cookies.json. It only reads public content you're already allowed to view.

---

🚀 Quick Start

# Scrape a single reel from the command line
python scraper.py "https://www.instagram.com/reel/CxXYZ123/"

# …or many reels from a file (one URL per line)
python scraper.py -f urls.txt -o export.csv

# Launch the desktop GUI
python gui.py

---

💻 Usage

1. Desktop GUI

python gui.py

• Click Login (optional but recommended — improves success rate).
• Paste one reel URL per line into the box (or Ctrl+A to select all).
• Drag the Workers slider, then click Scrape.
• Watch results appear in the table.
• Right‑click a row to Copy URL or Open Reel.
• Export to CSV / Excel / JSON, or Open results folder.

The last results are auto‑saved to results/_last_results.json.

2. Command Line (CLI)

python scraper.py [URL ...] [options]

Flag	Default	Description
urls	—	One or more reel URLs (positional).
-f, --file	—	Text file with one reel URL per line.
--login	off	Open a browser to log in interactively (QR).
--import-cookies FILE	—	Import an EditThisCookie JSON export.
--clear-session	off	Delete the saved storage_state.json.
--headless	off	Run the browser without a window.
-w, --workers	3	Number of concurrent scrape threads.
--delay	2.0	Seconds to wait between requests.
--state	storage_state.json	Path for the saved session.
-o, --output	reels_results.csv	Output CSV path.
--no-profiles	off	Skip auto‑fetching owner follower data.

# Headless, 5 workers, 1s delay, no profile enrichment
python scraper.py -f reels.txt -w 5 --delay 1 --headless --no-profiles -o out.csv

3. MCP Server (for AI agents)

Reelminner ships an MCP (Model Context Protocol) server so an AI client can drive it.

python mcp_server.py            # stdio transport

Configure your MCP client (.mcp.json is included in the repo):

{
  "mcpServers": {
    "reelminner": {
      "command": "python",
      "args": ["mcp_server.py"],
      "cwd": ".",
      "env": { "RMIN_HEADLESS": "true" }
    }
  }
}

Tools exposed (5, stable):

Tool	Signature	Purpose
scrape_reels	(urls, workers, delay, headless, with_profiles)	Run a scrape job.
get_status	()	Current progress / last result summary.
import_cookies	(json_path)	Load cookies from an EditThisCookie file.
stop_scrape	()	Stop the running job.
export_results	(path, fmt)	Export to csv / json / xlsx.

Environment overrides: RMIN_HEADLESS, RMIN_WORKERS, RMIN_DELAY, RMIN_WITH_PROFILES.

4. Python API

from scraper import Reelminner, write_csv

scraper = Reelminner(workers=3, delay=2.0, headless=True)
rows, report = scraper.scrape(
    ["https://www.instagram.com/reel/CxXYZ123/"],
    with_profiles=True,
)
write_csv(rows, "out.csv")

for r in rows:
    print(r["username"], r["followers"], r["likes"], r["status"])

Key members of Reelminner:

• scrape(urls, with_profiles=True) → (rows, report)
• login() — interactive login
• has_session() / save_cookies_from_file(path) / clear_session()
• write_csv(rows, path), export_json(rows, path), export_excel(rows, path)
• normalize_reel_url(url) — public helper
• csv_columns — the ordered list of output fields
• DEFAULT_STATE_FILE — default storage_state.json

---

📊 Output format

Each reel becomes one row. The full CSV schema (scraper.csv_columns):

Column	Description
idx	Row index.
username	Reel owner handle (e.g. natgeo).
followers	Owner follower count (may be follower_min–follower_max).
full_name	Owner display name.
bio	Owner biography text.
is_verified	True / False.
reels_count	Number of reels on the owner profile.
profile_url	Link to the owner profile.
reel_url	Canonical reel URL.
reel_id	Instagram reel shortcode / ID.
caption	Reel caption text.
upload_date	Post timestamp.
views	Play / view count.
likes	Like count.
comments	Comment count.
video_url	Direct video file URL.
thumbnail	Thumbnail image URL.
music_title	Audio track title.
music_artist	Audio artist.
music_id	Audio / music ID.
scrape_ts	When this row was scraped (ISO timestamp).
status	OK · PARSED_PARTIAL · FAILED · NO_DATA · BLOCKED · RATE_LIMITED.

---

⚙️ Configuration

Cookies / session

• Log in with python scraper.py --login (saves storage_state.json).
• Or export cookies from your browser via the EditThisCookie extension and run python scraper.py --import-cookies cookies.json.

Environment variables (used by MCP server & CLI defaults)

Variable	Effect
RMIN_HEADLESS	true/false — run browser headless.
RMIN_WORKERS	Default worker count.
RMIN_DELAY	Default delay between requests (seconds).
RMIN_WITH_PROFILES	true/false — auto‑enrich owner profiles.

A template is provided: copy mcp.env.example → mcp.env to override MCP defaults.

---

🗂️ Project structure

reelminner/
├── scraper.py          # Core engine: Reelminner + CLI
├── gui.py              # Tkinter desktop application
├── parsers.py          # Pure extraction helpers (HTML/JSON/music/regex)
├── mcp_server.py       # MCP server (5 tools for AI agents)
├── theme.py            # Dark‑theme styling for the GUI
├── build_exe.py        # PyInstaller build script
├── Reelminner.spec  # PyInstaller spec (one‑file EXE)
├── run_qa.py           # End‑to‑end QA harness with data‑quality gates
├── requirements.txt    # Runtime dependencies
├── requirements-dev.txt# Dev / test dependencies
├── mcp.env.example     # MCP env template
├── .mcp.json           # MCP client configuration
├── assets/             # Icons (icon.ico)
├── docs/               # SKILL.md, E2E test/fix plan
├── skills/             # Agent skill definition
├── tests/              # pytest suite + corpus.txt
└── results/            # Scrape outputs (git‑ignored)

---

🧪 Testing & QA

# Unit / integration tests
pytest -q

# End‑to‑end data‑quality run (uses your saved session)
python run_qa.py                 # full run over tests/corpus.txt
python run_qa.py --quick         # 1 URL, headless, fast iteration
python run_qa.py --url <reel>    # custom single URL
python run_qa.py --report-only   # show last qa_report.json

The QA harness enforces gates such as parsed‑rate, verified‑rate, non‑empty‑rate, blocked‑rate, and max runtime, and writes results/qa/qa_report.json + qa_results.csv.

---

📦 Building a standalone EXE

On Windows, produce a portable .exe (no Python needed by end users):

pip install pyinstaller
python build_exe.py

Output: dist/Reelminner.exe (one‑file build via Reelminner.spec).

---

⚠️ Legal & ethical disclaimer

Reelminner is provided for educational and authorized/personal use only.

• Scraping Instagram may violate its Terms of Service. Use it only on content you own or are permitted to access.
• Respect rate limits (--delay, fewer --workers) and do not use it for spam, harassment, or commercial bulk extraction.
• You are responsible for how you use this tool and for complying with applicable laws (incl. GDPR / privacy regulations) in your jurisdiction.
• The authors are not affiliated with Instagram/Meta and accept no liability.

---

🆘 Troubleshooting & FAQ

playwright says the browser isn't installed / pages won't open → Make sure you ran both pip install -r requirements.txt and playwright install chromium. Without the Chromium download nothing will launch.

Most fields are empty, or I get BLOCKED / RATE_LIMITED → Log in (python scraper.py --login) or import cookies, then slow down: --delay 4 and fewer workers (-w 1). Instagram throttles anonymous/unauthenticated traffic hardest, so an authenticated session is the single biggest success factor.

A reel returns NO_DATA → The post may be private, deleted, or region‑locked, or Instagram served a login wall. Try again with a logged‑in session.

The GUI window won't open or fonts look wrong → The GUI uses Python's built‑in tkinter. On Windows it's most polished. On Linux/macOS install the Tk package if the window fails to launch (e.g. sudo apt install python3-tk).

ModuleNotFoundError when I run a script → You're likely outside the repo or its virtual environment. cd into the project folder and activate the venv (.venv\Scripts\activate on Windows, source .venv/bin/activate on macOS/Linux) before running python scraper.py.

How do I scrape lots of reels at once? → Put one URL per line in a text file and run python scraper.py -f urls.txt -o out.csv.

Can an AI agent use this? → Yes — run python mcp_server.py and point any MCP client (Claude Desktop, Cursor, etc.) at the included .mcp.json. See MCP Server.

---

🤝 Contributing

1. Fork the repo and create a feature branch.
2. pip install -r requirements-dev.txt
3. Add/adjust tests in tests/; run pytest and python run_qa.py --quick.
4. Open a pull request describing the change and the QA result.

---

📄 License

Released under the MIT License — see LICENSE.

---

🏷️ Name

The project's final public name is Reelminner ("Reel miner"). Earlier internal codenames have been retired. If you fork it you can rename it to anything you like — just update the title in gui.py and this README.

<p align="center">Made with ❤️ for the open‑source scraping community.</p>