<div align="center">

📄 MarkIndex MCP

Enterprise Document Intelligence Server

Turn PDFs, Word docs, Markdown, websites, and YouTube transcripts into a local-first MCP knowledge base that LLMs can search, read, and navigate by stable section IDs — no vector database required.

If this helps your MCP/RAG workflow, please consider starring the repo ⭐

</div>

Key Advantages:

• 🏠 Local-first architecture
• 🚫 No vector DB required
• 🔑 Stable section IDs
• 📖 Section-level search/read/navigation
• 🤖 Works seamlessly with Claude Desktop & MCP clients
• ⚡ Built on Microsoft MarkItDown

---

Preview

Here’s what MarkIndex MCP looks like when an LLM searches and navigates a local policy document.

Server startup

Search result

Stable outline IDs

Read by section ID

Save outputs

---

✨ Features

Capability	Description
📥 Universal Ingestion	PDF, Word, Excel, PowerPoint, HTML, TXT, Markdown, URLs
🎬 YouTube Transcripts	Auto-download and index video transcripts with time-chunking
📂 Batch Directory Scan	Ingest all supported files from a directory in one call
🌳 Hierarchical Parsing	Detects #, SECTION, CHAPTER, APPENDIX, numbered, Roman, and timestamp headers
🔍 TF-IDF Search	Relevance-ranked full-text search with regex support and context snippets
📖 Paginated Reading	Character-level pagination for reading large sections without overflow
🧭 Tree Navigation	Parent, previous, next sibling traversal for sequential reading
📝 Extractive Summaries	Term-frequency sentence scoring for quick section overviews
💾 Persistent Cache	Markdown files with JSON frontmatter — human-readable, git-friendly

---

⚙️ How It Works: The 3-Folder Secret System

MarkIndex utilizes an organized, self-updating knowledge architecture:

1. raw/: Drop your source materials here (PDFs, Word documents, HTML, etc.). The server reads these files but never alters them.
2. wiki/: The server processes the raw files and structures them into cross-linked Markdown pages (one per document). It also generates a master index.md file that acts as a crawlable map, allowing the LLM to efficiently fetch context without wasting tokens.
3. outputs/: This folder automatically saves the results, reports, or plans generated every time you ask the LLM to write something based on your knowledge base.

By implementing this architecture, you essentially build a self-updating, personal consultation engine tailored to your exact data and files.

---

⚖️ Vector RAG vs. MarkIndex

How does our MarkIndex methodology compare to traditional Vector Database RAG?

Feature	Vector RAG	MarkIndex RAG
Context Preservation	4/10	10/10
Setup Complexity	3/10	9/10
Cost to Run	5/10	10/10
Sequential Reading	2/10	10/10
Token Efficiency	3/10	9/10
Fuzzy Semantic Match	9/10	6/10
Total Score	26/60	54/60

MarkIndex excels by preserving the original document hierarchy and allowing the LLM to paginate through full, unbroken sections, rather than receiving fragmented, out-of-context vector chunks.

Why MarkIndex RAG is Different:

1. Hierarchy vs. Chunks: Traditional Vector RAG chops documents into arbitrary 500-token chunks, destroying the author's intended structure. MarkIndex parses the actual headers (#, Chapter 1, etc.) to create a navigable tree with stable, unique section IDs.
2. Full Context: When an LLM asks MarkIndex for a section, it gets the entire section, exactly as it was written, rather than a few stitched-together vector matches that lack surrounding context.
3. No Expensive Embeddings: Vector RAG requires passing every document through an embedding model (like OpenAI text-embedding-ada-002), which costs time and API credits. MarkIndex uses an ultra-fast, local, pure-Python N-Gram TF-IDF engine for advanced multi-word lexical search.
4. Stable IDs & Context: MarkIndex tracks document paths deterministically (chapter-1-summary-2) allowing the LLM to easily distinguish between duplicate subheadings. When an LLM asks MarkIndex for a section by ID, it gets the entire section.
5. Token Efficiency: Vector RAG blindly dumps 5 to 10 disjointed chunks (2,500+ tokens) into the prompt. MarkIndex feeds the LLM a tiny structural map (index.md), and the LLM only fetches the specific, highly-relevant section it needs, drastically reducing token waste and API costs.
6. LLM Agency: With MarkIndex, the LLM acts like a human reader. It can read the Table of Contents, search for keywords, jump to a specific section, and then navigate to the "next" or "previous" sections.

Architecture

MarkIndex uses a robust "3-Folder Secret System" for enterprise knowledge management:

• raw/: Your original, untouched source documents (PDFs, Word docs, etc.).
• wiki/: The LLM's internal representation, stored as hierarchical Markdown files with JSON frontmatter.
• outputs/: Where the LLM automatically saves the persistent reports and answers it generates for you.

Note: You can strictly control whether the LLM is allowed to access files outside the raw/ directory via the MARKINDEX_ALLOW_EXTERNAL_FILES=true/false setting.

markindex-mcp/
├── markindex/                       # Python package
│   ├── __init__.py                  # Version & metadata
│   ├── __main__.py                  # python -m markindex
│   ├── config.py                    # Centralized Settings dataclass
│   ├── logger.py                    # Structured logging
│   ├── exceptions.py                # Custom exception hierarchy
│   ├── server.py                    # FastMCP server & lifecycle
│   ├── core/                        # Business logic
│   │   ├── parser.py                # Hierarchical document parser
│   │   ├── search.py                # TF-IDF ranking engine
│   │   ├── summarizer.py            # Extractive summarization
│   │   └── storage.py               # Frontmatter serialization & I/O
│   └── tools/                       # MCP tool definitions
│       ├── ingest.py                # Ingestion tools
│       ├── query.py                 # Querying tools
│       ├── navigate.py              # Navigation tools
│       └── manage.py                # Management tools
├── tests/                           # Test suite
├── pyproject.toml                   # PEP 621 packaging
├── requirements.txt                 # Dependencies
├── raw/                             # [NEW] Drop your source files here
├── wiki/                            # [NEW] Auto-generated markdown & master index.md
└── outputs/                         # [NEW] Claude's generated reports and summaries

---

⏱️ 30-Second Demo

Here are example MCP tool calls an LLM can make to process documents efficiently:

# 1. Ingest document (Place files in raw/ first unless MARKINDEX_ALLOW_EXTERNAL_FILES=true)
ingest_document("raw/company_policy.pdf")

# 2. Search for relevant sections using fast TF-IDF
search_sections(doc_id="doc_xyz123", query="vehicle compensation", limit=3)

# 3. Read specific section with full surrounding context
read_section(doc_id="doc_xyz123", section_title="vehicle-claims-compensation")

# 4. Navigate sequentially through the document
get_adjacent_sections(doc_id="doc_xyz123", section_title="vehicle-claims-compensation")

# 5. Save the final report locally
save_to_outputs("vehicle_claims_summary.md", summary)

Example Output

When you search or read sections, MarkIndex returns clean JSON structures:

search_sections() Result:

{
  "success": true,
  "data": [
    {
      "section_title": "vehicle-claims-compensation",
      "snippets": ["...eligible for vehicle compensation...", "...$0.65 per mile..."],
      "score": 12.4
    }
  ],
  "error": null,
  "code": null
}

read_section() Result:

{
  "success": true,
  "data": "## Vehicle Compensation\n\nEmployees who use their personal vehicles for corporate travel are eligible for vehicle compensation.\nThe rate is $0.65 per mile.",
  "error": null,
  "code": null
}

---

🚀 Quick Start

Prerequisites

• Python 3.11+
• pip

Installation

# Clone the repository
git clone https://github.com/rajfazulhussain2008/markindex-mcp.git
cd markindex-mcp

# Create a virtual environment
python -m venv venv
venv\Scripts\activate       # Windows
# source venv/bin/activate  # macOS/Linux

# Install dependencies
pip install -r requirements.txt

# Optional: YouTube transcript support
pip install youtube-transcript-api

Claude Desktop Configuration

Add the following to your claude_desktop_config.json (ensure you use absolute paths, install dependencies first, and restart Claude after saving):

{
  "mcpServers": {
    "markindex": {
      "command": "python",
      "args": ["-m", "markindex"],
      "cwd": "/absolute/path/to/markindex-mcp"
    }
  }
}

---

🔧 Configuration

All settings are managed via environment variables (prefix: MARKINDEX_):

Variable	Default	Description
MARKINDEX_RAW_DIR	./raw	Source materials directory
MARKINDEX_WIKI_DIR	./wiki	Processed markdown & master index directory
MARKINDEX_OUTPUTS_DIR	./outputs	AI generated reports directory
MARKINDEX_LOG_LEVEL	INFO	Log verbosity: DEBUG, INFO, WARNING, ERROR
MARKINDEX_ALLOW_EXTERNAL_FILES	false	Enable access outside raw/ directory
MARKINDEX_MAX_FILE_MB	50	Maximum file size for local/URL downloads

Copy .env.example → .env and customize as needed.

---

📚 Tool Reference

All tools return a consistent standard dictionary: {"success": true/false, "data": ..., "error": null, "code": null}

Core Tools

Tool	Description
ingest_document(filepath)	Download a URL or ingest a local file (strict size/type safety constraints).
ingest_directory(dir_path)	Recursively ingest a whole folder.
list_documents()	View all ingested docs.
delete_document(doc_id)	Completely purge a document from memory and disk.

LLM Exploration Tools

Tool	Description
get_document_outline(doc_id)	View the document's structure, titles, stable IDs, and sizes.
search_sections(doc_id, query)	Find specific keywords/regex using N-Gram TF-IDF engine.
read_section(doc_id, section_id)	Fetch the full markdown content of a section.
get_adjacent_sections(...)	Read the parent, previous, or next section.
summarize_section(...)	Generate an extractive summary of a huge section.

Management Tools

Tool	Description
list_documents()	List all ingested documents
delete_document(doc_id)	Delete a document from index and cache
save_to_outputs(filename, content)	Save AI-generated reports to the outputs/ folder
get_server_status()	Get the server's version, uptime, and memory statistics

---

🛡️ Security & Troubleshooting

Strict Path Security

MarkIndex enforces strict path traversal mitigation. By default, MARKINDEX_ALLOW_EXTERNAL_FILES=false. You cannot ingest local files outside of the raw/ directory, nor save outputs outside of outputs/. Files ingested via local paths or URLs are heavily sanitized. The maximum file size limit is controlled by MARKINDEX_MAX_FILE_MB (default 50).

Resolving Duplicate Section IDs

If a document contains multiple headers with the exact same text (e.g. ## Summary), MarkIndex assigns them deterministic, stable IDs by appending numerical counters (summary, summary-2, summary-3). When calling read_section or get_adjacent_sections, always use the exact section ID provided by get_document_outline or search_sections to avoid ambiguity.

Missing Any or dict Type Hint Errors

If you are upgrading from 1.x to 2.x, ensure you have installed the exact 2.0.0 version. MarkIndex 2.0.0 uses Python 3.11+ strictly typed ToolResponse models to guarantee clean JSON structures for all LLM tools.

Common Errors

Error Code	Meaning & Resolution
ACCESS_DENIED	You tried to ingest a local file outside the raw/ directory. Move the file into raw/ or set MARKINDEX_ALLOW_EXTERNAL_FILES=true.
FILE_TOO_LARGE	The file exceeds the MARKINDEX_MAX_FILE_MB setting (default 50MB).
TEXT_TOO_LARGE	The raw text passed to ingest_text exceeds MARKINDEX_MAX_TEXT_CHARS.
SECTION_NOT_FOUND	You requested a section title that doesn't exist. Check the suggestions in the error message or use get_document_outline().

---

🧪 Testing & Development

See our Contributing Guide to get started!

# Run the test suite
python -m pytest tests/ -v

---

📄 License

This project is licensed under the MIT License.

<div align="center"> Built with ❤️ by Rajmohamed H

Recommended Topics: mcp, model-context-protocol, rag, document-ai, document-intelligence, markitdown, python, llm-tools, claude, ai, knowledge-base, tf-idf, markdown </div>