Dataset-Pipeline-MCP

An MCP server that turns "find me a dataset" into "find me a dataset AND a correct, domain-aware preprocessing script for it" — callable from any MCP client (Claude Code, Claude Desktop) without leaving your terminal.

Why this one, specifically

Kaggle ships an official MCP server, and there are several community HuggingFace/Kaggle MCP connectors already. They cover search, download, and (in one case) an EDA-notebook prompt. None of them detect what kind of data a dataset actually is and generate a matching preprocessing pipeline and time-series/sensor data in particular gets silently treated as generic tabular data by every generic tool, which means wrong resampling, ignored sensor drift, and no windowing.

This server's actual job is: search → detect domain → generate a correct, runnable preprocessing script, with time_series_sensor as a first-class domain alongside tabular, nlp_text, image, and audio.

Tools

Tool	Description
search_datasets(query, max_results, include_kaggle)	Searches HuggingFace Hub (no auth needed) and optionally Kaggle (needs credentials). Annotates every result with a lightweight detected domain.
detect_domain(text, tags)	Classifies free text/tags into a domain, with a confidence score and the signals that were matched — not a black box.
generate_preprocessing_pipeline(dataset_id, domain, source, description_hint)	Renders a complete, runnable Python script. domain="auto" triggers metadata lookup + detection.
find_related_papers(topic, max_results)	arXiv search for preprocessing/methodology context.

Plus one resource (domains://catalog) and one prompt (dataset_report) to demonstrate full MCP surface coverage, not just tools.

Repository layout

server.py                    <- FastMCP wiring: tools, resource, prompt
core/
  domain_detector.py          <- pure heuristic classifier, fully unit-tested
  templates.py                 <- one preprocessing script generator per domain
connectors/
  huggingface.py               <- public search, optional token
  arxiv_search.py               <- public search, no auth
  kaggle_connector.py           <- optional, requires user-supplied credentials
scripts/
  verify_live.py               <- live end-to-end check against real APIs
tests/                        <- pytest suite, network-independent by design
Dockerfile
.mcp.json                     <- project-level Claude Code config
.env.example
mcp-config.example.json
requirements.txt / requirements-dev.txt

core/ has zero network dependencies by design — it's the part that has to be correct every time, so it's the part that's cheap to test exhaustively. connectors/ is where the world can fail, so every connector fails loudly and specifically, but never silently or fatally.

Setup

git clone https://github.com/pranjalisr/dataset-pipeline-mcp.git
cd dataset-pipeline-mcp
python -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate
pip install -r requirements.txt

No credentials are required for basic use — HuggingFace dataset search is public. Copy .env.example to .env and fill in HUGGINGFACE_TOKEN / Kaggle credentials only if you want higher rate limits or Kaggle search. .env is gitignored — double check with git check-ignore -v .env before committing if you're ever unsure.

Running generated scripts locally

generate_preprocessing_pipeline produces a script that depends on pandas, numpy, and scipy — these are not required to run the MCP server itself (the server only generates this code as text, it never imports these libraries), so they're kept in a separate scripts/requirements.txt rather than the top-level one:

pip install -r scripts/requirements.txt

Running it standalone

python server.py

This starts the server on stdio, which is how MCP clients launch local servers. Running it directly just makes it sit and wait for a client to connect — use one of the methods below to actually exercise it.

Verifying live network calls work

The automated test suite intentionally does not depend on live HuggingFace/arXiv/Kaggle access, so it passes even offline. To confirm the real network calls work end-to-end:

python scripts/verify_live.py
python scripts/verify_live.py --query "wearable sensor human activity recognition"

This calls the tools through the real MCP protocol layer for every supported domain plus a live Kaggle check when credentials are configured, and prints a PASS/FAIL summary with the resolved domain and confidence for each — including a hard failure if a result comes back with zero confidence, since a generated script with no real domain signal isn't trustworthy even though it's technically valid Python.

Interactively testing tools with MCP Inspector

npx @modelcontextprotocol/inspector python server.py

Prints a local URL with a UI listing all tools/resources/prompts where you can fill in arguments and see live JSON responses.

Running with Docker

docker build -t dataset-pipeline-mcp .
docker run -i --rm dataset-pipeline-mcp
npx @modelcontextprotocol/inspector docker run -i --rm --env-file .env dataset-pipeline-mcp

Connecting to Claude Code

This repo includes a project-level .mcp.json, so opening it in Claude Code prompts you to trust the dataset-pipeline server automatically. To wire it up manually instead:

claude mcp add --transport stdio dataset-pipeline -- python /absolute/path/to/dataset-pipeline-mcp/server.py

Once connected, just ask naturally — e.g. "find me a wearable sensor dataset for activity recognition and give me a preprocessing pipeline for it" — and the model decides which tools to call and in what order; you never name a tool directly.

Testing

pip install -r requirements-dev.txt
pytest tests/ -v

The bug hunt

This project's test suite didn't just check coverage gaps — every bug below was found by actually running the server against live data, a real Docker container, or a real Claude Code session, not by reading the code and assuming it was correct.

1. Lazy-generator exception escape. huggingface_hub.list_datasets() returns a lazy generator — the HTTP request only fires on iteration, so wrapping just the call in try/except let real API failures escape uncaught. Fixed by materializing the generator inside the try block.

2. HuggingFace's search param is a repo-name substring match, not full text search. A natural-language query like "human activity recognition sensor" returned zero results on a 200 OK — matching datasets existed, but not with that literal string in their name. Fixed with a keyword-fallback search plus relevance ranking.

3. Domain detection was reading empty YAML frontmatter. dataset_info().cardData is the README's front matter, which authors almost never fill in — the real description lives in the Markdown body, a separate fetch (DatasetCard.load().text). Detection was running on empty input for most real datasets until this was fixed.

4. Single-keyword overconfidence. The confidence formula (winning_score / total_score) reports 1.0 whenever only one domain has any signal — even a single ambiguous keyword. A wearable video dataset scored 100% confidence for time_series_sensor off the word "wearable" alone. Fixed with an evidence floor that dampens confidence when total signal strength is thin, verified to leave strong multi-signal matches untouched.

5. Kaggle SDK crash bug. import kaggle calls sys.exit(1) internally when credentials aren't recognized — and SystemExit is not caught by except Exception (it inherits from BaseException). A single bad Kaggle token could have crashed the entire server process, not just failed that one call. Confirmed against the real installed package and fixed by explicitly catching SystemExit at every Kaggle API boundary.

6. Image domain had no standalone keyword. Unlike tabular/nlp/ audio, which all have their own name as a strong signal, image only classified correctly by accident (via an unrelated "x-ray" match) on a real chest X-ray dataset. Fixed by adding "image" itself as a signal.

7. Blank HUGGINGFACE_TOKEN breaks auth. .env.example shows HUGGINGFACE_TOKEN= (blank) as a template — but os.environ.get() returns "" for a set-but-empty variable, not None, so an empty string was passed as a literal bearer token (Illegal header value b'Bearer ') instead of being treated as "no token." Fixed in the HuggingFace API client setup.

8. Client-side paste artifacts. A real MCP Inspector session had its own placeholder hint text (query: "wearable sensor") submitted literally instead of being replaced, polluting search results with unrelated matches on the stray word "query." Fixed with input sanitization that strips a recognized <label>: prefix and one layer of fully-wrapping quotes, verified not to touch genuine queries that happen to contain those words ("search query logs dataset" passes through unchanged).

Every fix above shipped with a regression test that reproduces the original failure, not just a check that the happy path still works.

Proof it works end-to-end

Beyond unit tests, this was verified through the full real stack: local stdio, live HuggingFace search across all five domains, live arXiv search, a live Kaggle call (crash-fixed and confirmed safe on invalid credentials), a built and running Docker container connected via MCP Inspector, and finally a real Claude Code session — asked in plain English, with no tool names given — that correctly chained search_datasets → generate_preprocessing_pipeline, found DiFronzo/Human_Activity_Recognition, generated the time_series_sensor template, then independently discovered the template's assumptions didn't match that dataset's real file layout (separate accelerometer/gyroscope files + a label-segment file, no timestamps) and rewrote the script to handle it correctly — producing 3,289 labeled windows across 5 balanced activity classes with a proper subject-wise train/test split.

Known limitations

• Domain detection is keyword/heuristic-based, not a trained classifier — transparent and fast, but a genuinely ambiguous or sparsely-described dataset can be misclassified. detect_domain always returns its confidence and matched signals so a caller can tell when to double-check.
• search_datasets's per-result domain field is a lightweight, free estimate (id words + whatever description/tags came back in the list response) — generate_preprocessing_pipeline(domain="auto") does a deeper per-dataset fetch and is the one to trust for a real classification.
• Kaggle search requires the user's own API credentials (KAGGLE_API_TOKEN recommended); this server never requests, stores, or proxies them beyond reading environment variables. The full success path with a genuinely valid token hasn't been verified in this project's own testing — only the credential-missing and credential-invalid paths have live coverage.
• Preprocessing templates are strong starting points, not final pipelines — they're meant to be read and adapted (target columns, window sizes, actual file formats), not run blindly in production.