AI Research Paper Assistant

A Retrieval-Augmented Generation (RAG) system that answers questions about foundational AI/ML research papers, with an agentic reasoning layer, automated evaluation, and an MCP (Model Context Protocol) server so any MCP-compatible LLM client can use it as a tool.

---

What it does

Ask questions like:

• "What is the attention mechanism in Transformers?"
• "How does RAG combine retrieval with generation?"
• "Compare how the Transformer paper and ReAct paper approach reasoning"

The system retrieves relevant passages from three indexed papers — Attention Is All You Need, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, and ReAct: Synergizing Reasoning and Acting in Language Models — and generates a grounded answer, citing the source paper and page.

For complex, multi-part questions, an agent layer breaks the question into sub-questions, retrieves context for each, and synthesizes a combined answer — rather than relying on a single retrieval pass.

---

Architecture

┌─────────────┐
│   papers/   │  Raw PDFs (downloaded from arXiv)
└──────┬──────┘
       │  ingest.py: load → chunk → embed
       ▼
┌─────────────┐
│  chroma_db/ │  Vector store (291 chunks, all-MiniLM-L6-v2 embeddings)
└──────┬──────┘
       │
       ├─────────────────┬─────────────────────┐
       ▼                 ▼                      ▼
┌─────────────┐  ┌──────────────┐   ┌────────────────────┐
│ rag_chain.py│  │  agent.py    │   │  evaluation.py      │
│ Simple LCEL │  │  LangGraph:  │   │  RAGAS metrics:      │
│ retrieve→   │  │  plan→       │   │  faithfulness,        │
│ generate    │  │  retrieve→   │   │  answer relevancy,    │
│             │  │  synthesize  │   │  context precision    │
└──────┬──────┘  └──────┬───────┘   └─────────────────────┘
       │                │
       └───────┬────────┘
               ▼
      ┌─────────────────┐
      │  mcp_server.py   │  Exposes both as MCP tools
      │  + app.py (CLI)  │  for external LLM clients
      └─────────────────┘

Why this shape: all shared setup (embeddings, vector store connection, LLM client) lives in one place, src/config.py. Every other module imports from it instead of redefining it — so changing the embedding model or LLM provider means editing one file, not four.

---

Tech stack and why each piece was chosen

Component	Choice	Why
Orchestration	LangChain (LCEL)	Industry-standard way to compose retrieval + prompt + LLM into a pipeline
Agent framework	LangGraph	Lets the system plan multi-step retrieval instead of one fixed pass — needed for comparison-style questions
Vector store	Chroma	Local, free, simple to set up and inspect for a single-developer project
Embeddings	HuggingFace all-MiniLM-L6-v2	Runs locally on CPU, free, no API key or vendor lock-in — deliberate alternative to OpenAI embeddings
LLM	Groq (Llama 3.3 70B)	Free tier, very fast inference, good enough quality for this scope
Evaluation	RAGAS	Automated, LLM-judged scoring instead of just eyeballing answers
Tool exposure	MCP (Model Context Protocol)	Turns the pipeline into a callable tool for any MCP client (e.g. Claude Desktop), not just a terminal script

---

Evaluation results

Measured with RAGAS on a 3-question test set against known ground-truth answers:

Metric	Score	What it measures
Faithfulness	0.87	Does the answer stick to what's actually in the retrieved context, or hallucinate?
Answer Relevancy	0.93	Does the answer actually address the question asked?
Context Precision	0.94	Were the most relevant chunks ranked near the top of retrieval?

---

Project structure

ai-research-paper-assistant/
├── src/
│   ├── config.py       # Single source of truth: embeddings, vectorstore, LLM setup
│   ├── ingest.py        # PDF loading, chunking, embedding into Chroma
│   ├── rag_chain.py     # Simple retrieve-then-generate LCEL chain
│   ├── agent.py          # LangGraph agent: plan → retrieve → synthesize
│   └── evaluation.py    # RAGAS evaluation harness
├── mcp_server.py         # MCP tool wrapper around rag_chain and agent
├── app.py                 # CLI entry point (ingest / ask / agent / evaluate)
├── download_papers.py     # One-time script to pull papers from arXiv
├── requirements.txt
└── eval_results.txt        # Saved RAGAS output

---

Setup

# 1. Clone and enter the project
git clone <your-repo-url>
cd ai-research-paper-assistant

# 2. Create and activate a virtual environment
python -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Add your Groq API key (free at console.groq.com)
echo "GROQ_API_KEY=your_key_here" > .env

# 5. Download the papers and build the vector store
python download_papers.py
python app.py ingest

Known dependency issue: ragas==0.3.9 ships with a broken import (langchain_community.chat_models.vertexai, a module path that's since moved). If you hit ModuleNotFoundError on that import when running evaluate, it's a known upstream bug, not a problem with this code — open an issue or check for a patched ragas release. A workaround is to wrap that one import line in the installed package in a try/except ImportError, since this project doesn't use VertexAI at all.

---

Usage

python app.py ask         # Simple RAG: ask direct questions, one retrieval pass
python app.py agent       # Agentic RAG: handles multi-part/comparison questions
python app.py evaluate    # Re-run RAGAS scoring, saves to eval_results.txt
python mcp_server.py      # Start as an MCP server for external clients (e.g. Claude Desktop)