Ragify Docs MCP

ragify_docs_mcp is a small Model Context Protocol server that scrapes a documentation site, chunks the content, embeds it locally, and returns the most relevant text blocks for a query.

It is designed for retrieval over docs pages, API references, framework guides, and other public websites that you want to ask questions about from an agent or an MCP-aware client.

What it does

The server exposes one tool:

• ragify_docs_mcp(url: str, query: str = "What is this website about?") -> str

Given a starting URL, the tool recursively crawls linked pages, splits the text into chunks, embeds the chunks with a local sentence-transformers model, and retrieves the most relevant passages for your query.

Requirements

• Python 3.12 or newer
• An internet connection for scraping target sites and downloading the embedding model the first time
• uv is recommended for running the packaged entrypoint with uvx

Installation

From the project root:

uv sync

If you prefer standard pip tooling, install the project dependencies from requirements.txt or build an editable install from the source tree.

Run the MCP server

The package exposes the ragify_docs_mcp command:

ragify_docs_mcp

That starts the FastMCP server over stdio.

You can also run the published command through uvx, which is the same approach used in the client example:

uvx ragify_docs_mcp

Use it from the example client

The file client.py shows how to connect to the server, list its tools, and call it from a LangChain agent.

It uses this server configuration:

client = MultiServerMCPClient(
	{
		"ragify_docs_mcp": {
			"transport": "stdio",
			"command": "uvx",
			"args": ["ragify_docs_mcp"],
		}
	}
)

Example: list available tools

tools = await client.get_tools()

print("\nAvailable tools:")
for tool in tools:
	print(tool.name)

Example: use the tool through an agent

from langchain_mcp_adapters.client import MultiServerMCPClient
from langchain.agents import create_agent
import asyncio


async def main():
	client = MultiServerMCPClient(
		{
			"ragify_docs_mcp": {
				"transport": "stdio",
				"command": "uvx",
				"args": ["ragify_docs_mcp"],
			}
		}
	)

	tools = await client.get_tools()

	agent = create_agent(
		model="ollama:llama3.2:latest",
		tools=tools,
		system_prompt="You are a helpful assistant.",
	)

	response = await agent.ainvoke(
		{
			"messages": [
				{
					"role": "user",
					"content": "Summarize the docs at https://docs.example.com and tell me how authentication works.",
				}
			]
		}
	)

	print(response["messages"][-1].content)


if __name__ == "__main__":
	asyncio.run(main())

Tool behavior

The tool is intentionally simple:

1. It starts from the URL you provide.
2. It recursively loads pages under that site.
3. It extracts visible text with BeautifulSoup.
4. It splits the scraped text into chunks.
5. It embeds the chunks locally using sentence-transformers/all-MiniLM-L6-v2.
6. It returns the top matching chunks for your query.

The return value is plain text, already concatenated for downstream agents.

When to use it

This server is useful when you want an agent to answer questions grounded in a documentation site without manually copy-pasting pages.

Typical requests include:

• "What does this library do?"
• "Find the auth configuration options in this docs site."
• "Show me the code example for the retry policy."
• "Summarize the sections about installation and setup."

Limitations

• The scraper only sees content reachable from the starting URL.
• Sites that heavily rely on client-side rendering may not scrape cleanly.
• Very large sites can take time to crawl because the server embeds content in memory for the current request.
• The local embedding model must be downloaded the first time the tool runs.

Project layout

• client.py - Example LangChain client that loads the MCP server and uses the tool.
• src/ragify_docs_mcp/main.py - CLI entrypoint that starts the server.
• src/ragify_docs_mcp/server.py - MCP tool implementation.
• pyproject.toml - Package metadata, dependencies, and script entrypoint.

Development notes

The server is exposed as a standard Python package script named ragify_docs_mcp. The project uses the src/ layout, so local edits should be made under src/ragify_docs_mcp/.

If you change the tool signature or add new tools, update this README so the example client stays in sync.

Troubleshooting

If uvx ragify_docs_mcp fails, check that uv is installed and available on your PATH.

If the tool returns empty context, verify that the URL is reachable and that the site exposes crawlable HTML rather than only rendered client-side content.

If the first request is slow, that is usually the embedding model download plus the crawl and indexing step.