datamgr — dataset registry for empirical research

A lightweight registry that catalogs your research datasets so an AI (or you) can discover them without reading every file. Each dataset is one folder with a manifest.yaml; a flat index (_registry.json) makes search cheap no matter how many datasets you have.

Architecture: one copy of logic in datamgr.core; the dm CLI and the MCP server are both thin front-ends over it.

Install (editable)

cd data-manage
pip install -e .

Quick start

# 1. point datamgr at a central warehouse (created if missing)
dm init-root D:/datasets

# 2a. import a raw data folder -- harvests variable names + labels from a .dta,
#     copies the file into a new clean-id folder under the warehouse
dm import "原始数据包" --id csmar_annual --name "CSMAR 上市公司年度财务"

# 2b. (or) scaffold an empty dataset and fill the manifest by hand
dm init csmar_annual --name "CSMAR 上市公司年度财务"

# 3. rebuild the index
dm refresh

# 4. discover
dm list                          # all datasets, one line each
dm show csmar_annual             # full manifest, rendered
dm search 资产负债率               # fuzzy-search across all variables/labels
dm variables --role control      # list all control variables available
dm variables --label 占比          # filter variables by label
dm stats csmar_annual --refresh  # compute rows/cols/n_firms, cache to manifest
dm verify                        # validate manifests + check file paths exist

# 5. pull data into an analysis working directory (backup + provenance)
dm export csmar_annual control_vars --to D:/my_paper
#   -> D:/my_paper/data/*.dta        (copied files, ready for `use "data/xxx.dta"`)
#   -> D:/my_paper/data_sources.txt  (where each file came from, for traceability)

Commands

command	purpose
dm init-root [PATH]	set / show the central warehouse location
dm init <id>	scaffold an empty dataset folder + manifest template
dm import [<folder>] [--id X]	build a manifest by harvesting variables from a .dta/.csv; with --id it copies data into a clean-id folder
dm refresh	rebuild the _registry.json index (reads manifests only, never data files)
dm list [--tag T]	list all datasets, one line each
dm show <id>	print a dataset's full manifest
dm search <query> [--tag T]	fuzzy-search datasets by variable name/label, name, description, tag
dm variables [--role R] [--label L]	list all variables across the index (for picking controls/instruments)
dm stats <id> [--refresh]	show cached stats, or recompute from the data file
dm export <id...> --to <dir>	copy dataset files into a working dir + record provenance
dm verify	validate manifests + check referenced files exist

The manifest (manifest.yaml)

This is the heart of the system. Fields:

field	purpose
id	dataset id, must equal the folder name
name	human-readable name
description	free text — searchable
version, source	provenance
unit_of_observation	granularity, e.g. 公司-年
time_span	[start, end] years
identifiers	{id, time} — fed straight to Stata xtset
variables	list; each has name/label/type/role
tags, dependencies	search facets & lineage
files	relpaths for raw/ pipeline/ processed
stats	cached summary (rows, cols, ...), computed at refresh
notes	any gotchas /口径变更

variables[].role ∈ {id, time, x, y, control, weight, instrument, other} is the key signal that lets an AI judge "can this dataset run the regression I want", which plain variable names can't convey.

Why it stays fast as datasets grow

• Search reads the in-memory index, never the folder tree at query time.
• The index stores only search-relevant fields (names, roles, tags, spans) — KB per dataset. A full manifest is loaded only by dm show / get_dataset.
• Dataset files (.dta, etc.) are read only when you explicitly refresh stats. list/search/show never touch them.

Layout

src/datamgr/
  config.py        warehouse root resolution
  core/
    manifest.py    schema + load/validate/save
    registry.py    scan -> _registry.json index
    search.py      fuzzy search over the index (rapidfuzz)
    stats.py       compute rows/cols/n_firms from a .dta/.csv (explicit --refresh only)
    importing.py   harvest variables from a .dta/.csv -> manifest scaffold
    exporting.py   copy data files to a working dir + provenance txt
  cli/main.py      `dm` command
  mcp/server.py    read-only MCP server for AI discovery (stdio)

MCP server (AI discovery)

A read-only MCP server lets an AI client (e.g. zcode) discover and inspect datasets without touching data files. Exposes 5 tools over stdio:

tool	purpose
list_datasets(tag)	all datasets (id/name/vars/rows/span/tags)
search_datasets(query)	fuzzy search across variables/labels/names
list_variables(role, label)	flat variable list for picking controls
get_dataset(id)	full manifest incl. absolute file paths (locate the .dta on disk)
get_dataset_stats(id)	cached stats only (never reads data files)

Run it directly, or register with an MCP-aware client:

# direct
python -m datamgr.mcp.server
# or via entry point
datamgr-mcp

ZCode workspace registration (.zcode/config.json):

{
  "mcp": {
    "servers": {
      "datamgr": {
        "command": "python",
        "args": ["-m", "datamgr.mcp.server"],
        "env": { "DATAMGR_ROOT": "D:/datasets" }
      }
    }
  }
}

The server is deliberately read-only: importing data and recomputing stats are done from the dm CLI by a human, never by the AI.

Deployment & daily use

See DEPLOYMENT.md for:

• deploying to another machine or MCP client (zcode / Claude / Cursor)
• daily workflow when new data arrives (double-click .bat scripts, no CLI needed)
• troubleshooting and a full migration checklist