AI Papers Helper

Search, index, and read academic AI/ML papers — from the terminal or directly inside your AI coding assistant.

AI Papers Helper crawls top-tier AI/ML conference proceedings, indexes them in a local SQLite database with FTS5 search, queries arxiv, and converts paper PDFs into clean markdown for in-depth reading. It ships as both a CLI (papers) and an MCP server (ai-papers-mcp), so you can use it standalone or wire it into Claude Code / any MCP-compatible client.

---

✨ Features

• 🌐 Conference crawling — Fetch papers from NeurIPS, ICML, ICLR, CVPR, ICCV, WACV, AAAI, IJCAI, MLSys, ACL, EMNLP with parallel workers and incremental crawl state.
• 🔎 Local search — SQLite + FTS5 with BM25 ranking (title-weighted) and relevance/date ordering. Missing-abstract penalty keeps results meaningful.
• 📡 arxiv search — Live arxiv search via web scraping (default) or the Atom API backend, with date filters (past 12 months, specific year, date range).
• 📄 PDF → Markdown — Parse papers through the MinerU API with content-addressable caching, so each PDF is only parsed once.
• 📑 TOC & grep — Extract a paper's table of contents and grep specific sections/equations from its full markdown.
• 🤝 MCP server — Expose everything as 4 tools to Claude Code or any MCP client: search the local library, search arxiv, get a paper's TOC, and grep paper content.
• 🧠 Smart resolution — Paper lookup falls back progressively: exact DB match → fuzzy match (SequenceMatcher, 0.8 threshold) → arxiv search.

---

📦 Installation

Requires Python 3.12+.

# Clone
git clone https://github.com/<your-org>/ai_papers_helper.git
cd ai_papers_helper

# Install as a global tool (exposes `papers` and `ai-papers-mcp` on PATH)
uv tool install --force .

This exposes two console scripts:

Command	Description
papers	The CLI app
ai-papers-mcp	The MCP server

Environment variables

Variable	Default	Description
MINERU_API_KEY	—	Required for PDF→markdown parsing. Get one at mineru.net.
ARXIV_BACKEND	web	arxiv backend: web (scraping) or api (Atom API).
ARXIV_MIN_INTERVAL	10.0	Minimum seconds between arxiv API calls (rate limiting).

All data lives under ~/.ai_papers_helper/:

~/.ai_papers_helper/
├── papers.db            # SQLite database (FTS5 index)
├── crawl_state.json     # Incremental crawl state
├── cache/               # HTTP response cache
├── arxiv_cache/         # arxiv result cache (24h TTL)
└── papers/              # Parsed markdown (content-addressed by URL hash)

---

🚀 Quick start

1. Initialize the database

papers init

2. Crawl conference papers

# Crawl everything new (incremental — skips already-crawled years)
papers update

# Crawl a specific conference and year
papers update --conference acl,emnlp --year 2024

# Force re-crawl a specific source/year
papers update --force --conference cvpr --year 2023

# Tune parallelism
papers update --workers 16

3. Search

# Search the local library (default: titles only)
papers search-library "diffusion model"
papers sl "graph neural network" --order-by date

# Titles + full abstracts
papers sl "transformer attention" --full-abs

# Paginate
papers sl "reinforcement learning" --page 2
papers sl "reinforcement learning" --from 31

# Search arxiv
papers search-arxiv "mixture of experts"
papers sa "vision transformer" --sort-by date --date-filter-by past_12
papers sa "llm" --date-filter-by specific_year --date-year 2024
papers sa "diffusion" --date-filter-by date_range --date-from 2024-01 --date-to 2024-06

4. Read a paper

# Get the full markdown (creates a /tmp/<title>.md symlink to the cached file)
papers content "Attention Is All You Need"

# Show the table of contents
papers content "Attention Is All You Need" --toc

---

🧩 Skill (for AI coding agents)

The repo ships a ready-made Agent Skills skill at papers-skill/SKILL.md. It teaches AI coding agents (pi, Claude Code, etc.) how to drive the papers CLI: when to prefer the local library vs arxiv, the search -> content -> grep workflow, and the full option reference.

The agent loads the skill on-demand when a task matches, then runs papers itself via the shell - no server process required. This is the lightest-weight way to let an agent search and read papers.

Install the skill

Point your agent at the papers-skill directory. For pi:

# Global (available in every project)
ln -s "$(pwd)/papers-skill" ~/.pi/agent/skills/papers

# Or project-level
mkdir -p .pi/skills && ln -s "$(pwd)/papers-skill" .pi/skills/papers

Skill vs MCP: The skill is just instructions (the agent runs the CLI via shell); the MCP server below exposes typed tools. The skill needs nothing running, the MCP server gives more structured tool calls - pick whichever fits your agent.

---

🤖 Using the MCP server

The same functionality is exposed as an MCP server for use inside Claude Code or any MCP-compatible client.

4 tools

Tool	Description
search_library_papers	Search the local indexed database by keywords.
search_arxiv_papers	Search arxiv for the latest papers, with date filters.
get_paper_toc	Get a paper's table of contents. Call this first before reading.
grep_paper_content	grep patterns in a paper's full markdown (e.g. read a whole section).

---

🏗 Architecture

src/ai_papers_helper/
├── cli.py                 # Typer CLI: init, search-library, search-arxiv, update, content
├── mcp_server.py          # FastMCP server exposing 4 tools
├── config.py              # Paths, env vars, page-size constants
├── core/
│   ├── models.py          # Pydantic v2: Author, Paper
│   └── database.py        # SQLite + FTS5 singleton, BM25 ranking, auto-sync triggers
├── crawler/
│   ├── base.py            # BaseCrawler ABC + parallel detail-page fetching
│   ├── http.py            # Shared requests.Session w/ retry + file cache
│   ├── state.py           # CrawlState (per-source crawled years, JSON)
│   ├── cvf.py             # CVPR / ICCV / WACV
│   ├── aaai.py            # AAAI
│   ├── ijcai.py           # IJCAI
│   ├── icml.py            # ICML URL helpers
│   ├── acl_anthology.py   # ACL / EMNLP (ACL Anthology)
│   └── json_api.py        # Generic JSON API crawler (NeurIPS, ICML, ICLR, MLSys)
├── search/
│   ├── library_search.py  # Local FTS5 search, relevance/date ordering
│   └── arxiv_search.py    # arxiv dispatcher (web vs api backend)
├── retrieval/
│   ├── resolver.py        # Progressive lookup: exact → fuzzy → arxiv
│   ├── parser.py          # MinerU API client (async polling, content-addressed cache)
│   ├── content.py         # Markdown TOC extraction + section slicing
│   └── lookup.py          # End-to-end: title → paper → markdown
└── helper/                # arxiv web/api internals, pagination, rate limiting

How search works

• Local library: FTS5 with porter unicode61 tokenizer. BM25 with title weight 10.0, abstract weight 1.0. Results with missing abstracts are penalized (* 0.9) so well-documented papers surface first.
• Date ordering: BM25 rank is bucketed into relevance tiers; within a tier, newer papers come first — so you don't lose relevance entirely.
• arxiv: Web scraping by default (no API key, gentler). Switch to the Atom API with ARXIV_BACKEND=api for query-syntax power (field prefixes, boolean operators).

How PDF reading works

1. Resolve the paper by title (DB exact → fuzzy → arxiv).
2. Resolve a PDF URL (paper.pdf_url, else arxiv lookup, backfilling the DB).
3. Send to MinerU; poll until done; download & unzip the result.
4. Cache under ~/.ai_papers_helper/papers/{sha256(url)[:16]}/full.md — content-addressed, so re-reads are instant.

---

🧪 Development

uv sync                 # install deps
pytest                  # run the full suite
pytest tests/test_database.py          # single file
pytest -k "fuzzy"                      # by name pattern

Tests use temp databases, mock network calls (patch.object(crawler, "_fetch_url", ...)), and HTML/JSON fixtures in tests/fixtures/. See CLAUDE.md for the full contributor guide.

Conventions

• Python 3.12+, from __future__ import annotations in every file
• No async; parallelism via ThreadPoolExecutor
• Standard-library sqlite3 (no ORM), requests for HTTP
• Logging via logging.getLogger(__name__)

---

📄 License

MIT