🧠 Second Brain

A personal AI-powered knowledge base. Ingest your PDFs, notes, bookmarks, code files, and YouTube videos — then chat with them using Claude.

---

What it does

• Ingest any document: PDFs, Markdown, DOCX, code files, web URLs, YouTube transcripts, Notion pages
• Search using hybrid retrieval — semantic vector search + keyword search, merged with a real ML re-ranker
• Chat with your documents through a web UI, answers come with source citations
• Expose your knowledge base as an MCP tool so Claude Desktop can search it mid-conversation

---

Quick start

1. Clone and install

git clone https://github.com/MBA009/RAG-MCP-system.git
cd RAG-MCP-system
pip install -r requirements.txt

2. Set up environment variables

Create a .env file in the project root:

ANTHROPIC_API_KEY=sk-ant-api03-...
EMBEDDING_BACKEND=local

EMBEDDING_BACKEND=local uses a free offline model (recommended to start). If you have an OpenAI key and want higher quality embeddings:

EMBEDDING_BACKEND=openai
OPENAI_API_KEY=sk-...

Optional:

NOTION_TOKEN=secret_...

3. Ingest your documents

# Single file
python cli.py ingest ~/Documents/notes.pdf

# Entire folder (recursive)
python cli.py ingest ~/Documents/notes/

# Web page
python cli.py ingest-url https://en.wikipedia.org/wiki/Retrieval-augmented_generation

# YouTube video (fetches transcript automatically, no API key needed)
python cli.py ingest-youtube https://www.youtube.com/watch?v=VIDEO_ID

# All Notion pages (requires NOTION_TOKEN in .env)
python cli.py ingest-notion

4. Start the web UI

python cli.py web

Open http://localhost:8000 in your browser.

---

Web UI features

• Chat with your documents — answers include clickable source citations
• Add sources from the sidebar: URL, YouTube, file upload, or Notion sync
• Hover over any source and click ✕ to remove it
• Header shows total docs and chunks indexed

---

All CLI commands

Command	What it does
python cli.py ingest <path>	Ingest a file or directory
python cli.py ingest-url <url>	Ingest a web page
python cli.py ingest-youtube <url>	Ingest a YouTube transcript
python cli.py ingest-notion	Sync all Notion pages
python cli.py list	List all ingested sources
python cli.py delete <hash>	Delete a source by its hash
python cli.py stats	Show total sources and chunks
python cli.py watch	Watch inbox folder and auto-ingest new files
python cli.py chat	Terminal chat interface
python cli.py web	Start the web UI
python cli.py serve	Start the MCP server for Claude Desktop

---

Claude Desktop integration (MCP)

Add to your Claude Desktop config:

Windows: %APPDATA%\Claude\claude_desktop_config.json Mac: ~/.config/claude/claude_desktop_config.json

{
  "mcpServers": {
    "second-brain": {
      "command": "python",
      "args": ["cli.py", "serve"],
      "cwd": "C:\\path\\to\\second-brain"
    }
  }
}

Restart Claude Desktop. Claude can now call search_brain, add_document, list_sources, and delete_source as tools mid-conversation.

---

Key configuration (config.py)

Setting	Default	Description
EMBEDDING_BACKEND	"local"	"local" or "openai"
CHUNK_SIZE	512	Tokens per chunk
TOP_K_FINAL	5	Results returned per query
CLAUDE_MODEL	claude-sonnet-4-...	Claude model used for chat

---

Supported file types

PDF, DOCX, Markdown, plain text, Python, JavaScript, TypeScript, Go, Rust, C, C++, Java, HTML, web URLs, YouTube, Notion, Obsidian vaults

---

Tech stack

Tool	Role
Qdrant	Local vector database
SQLite FTS5	Full-text keyword search
sentence-transformers	Embeddings + cross-encoder re-ranking
FastAPI	Web API
Anthropic Claude	Language model
MCP SDK	Claude Desktop tool integration