LinkedRun

LinkedRun is a persistent local Task–Artifact DAG executor exposed as an MCP server.

Tasks consume immutable artifacts and produce immutable artifacts. A new task may depend on outputs from tasks submitted in the same batch or from any earlier submission. The graph therefore grows incrementally instead of being bounded by one workflow run.

LinkedRun is deliberately a mechanism-only kernel. It does not know what training, evaluation, cell clustering, ARI, a model, or an experiment protocol means. It also does not predict resource requirements: the submitting Agent declares resources and LinkedRun trusts that declaration, subject only to machine-capacity limits and runtime reservation.

Why

The intended split is:

• task Agent: decides what to run and declares resource needs;
• project tooling: predicts resources, validates models, builds workflows, computes metrics;
• LinkedRun: persists dependencies, waits without busy polling, reserves declared resources, runs and cancels processes, commits artifacts, and records durable events.

Core model

Artifact -> Task -> Artifact
                \\-> Task -> Artifact

A data dependency is also an execution dependency. Pure ordering constraints are available through after when no artifact is consumed.

MCP tools

• submit_task
• submit_graph
• get_task
• list_tasks
• cancel_task
• retry_task
• list_artifacts
• get_artifact
• get_graph
• get_events
• watch_events
• resource_status

watch_events is a durable long-poll interface: clients resume from the last event_id, so they do not need tight polling loops. A future release can map execution handles onto the MCP io.modelcontextprotocol/tasks extension when host support is sufficiently common.

Install

pip install -e .

Python 3.11+ is required. LinkedRun targets MCP Python SDK v2 / MCP 2026-07-28.

Start

Persistent local HTTP service (recommended when several Agents/clients need the same graph):

export LINKEDRUN_HOME="$HOME/.linkedrun"
linkedrun --transport streamable-http --host 127.0.0.1 --port 8765

The MCP endpoint is http://127.0.0.1:8765/mcp.

For a host that manages the MCP process itself:

linkedrun --transport stdio

SQLite state and content-addressed artifacts are stored below LINKEDRUN_HOME.

Submit one task

Conceptually, an MCP call to submit_task looks like:

{
  "name": "train",
  "command": ["python", "train.py"],
  "outputs": {
    "model": "outputs/model.pt",
    "embedding": "outputs/embedding.zarr"
  },
  "resources": {
    "cpu_cores": 8,
    "memory_bytes": 34359738368,
    "gpu_count": 1,
    "gpu_mode": "exclusive"
  }
}

Commands are argv arrays, not shell strings. Use ["bash", "-lc", "..."] explicitly when shell semantics are required.

Same-submission dependencies

submit_graph supports local references:

{
  "tasks": [
    {
      "name": "train",
      "command": ["python", "train.py"],
      "outputs": {"embedding": "outputs/embedding.zarr"}
    },
    {
      "name": "cluster",
      "command": ["python", "cluster.py"],
      "inputs": {"embedding": "@train/embedding"},
      "outputs": {"clusters": "outputs/clusters.parquet"}
    },
    {
      "name": "ari",
      "command": ["python", "ari.py"],
      "inputs": {"clusters": "@cluster/clusters"}
    }
  ]
}

The complete batch is registered atomically after cycle detection.

Cross-submission dependencies

If an older training task has ID task_abcd and produced embedding, a task submitted later may use:

{
  "inputs": {
    "embedding": "task:task_abcd/artifact:embedding"
  }
}

A committed artifact can also be referenced directly:

artifact:art_abcd

The graph is therefore persistent and incremental: no top-level "workflow run" boundary is required.

Runtime contract

Before a task starts, LinkedRun creates a private attempt directory and sets:

LINKEDRUN_TASK_ID
LINKEDRUN_ATTEMPT_ID
LINKEDRUN_WORKDIR
LINKEDRUN_OUTPUT_DIR
LINKEDRUN_INPUT_<NAME>

Each input is a read-only-by-convention symlink to immutable content-addressed storage. Declared output paths must remain inside the attempt directory. On successful process exit, outputs are hashed and committed to the artifact store before the task becomes SUCCEEDED.

Task states

PENDING -> READY -> RUNNING -> SUCCEEDED
                       \\----> FAILED
PENDING --------------------> BLOCKED      (upstream failed/missing artifact)
PENDING --------------------> UNSCHEDULABLE (declared request exceeds machine capacity)
PENDING/RUNNING ------------> CANCELED

Retry increments a per-task generation. An old attempt may not commit after its generation becomes stale; this is the first implementation of LinkedRun's fencing rule.

Resource policy

LinkedRun does not infer resource usage. It accepts:

cpu_cores
memory_bytes
gpu_count
gpu_mode=exclusive
gpu_memory_bytes_hint
walltime_seconds
scratch_bytes

gpu_memory_bytes_hint is evidence/metadata only in v0.1. GPU scheduling is exclusive-device allocation. CPU and memory are reservation accounting; walltime_seconds is enforced. OS-level hard CPU/memory/scratch isolation is intentionally left for a later sandbox module rather than adding experiment-specific admission logic to the kernel.

Current v0.1 boundaries

Implemented:

• SQLite/WAL persistent task, attempt, dependency, artifact, and event state;
• atomic same-batch graph registration and cycle detection;
• cross-submission task/artifact references;
• background dependency scheduling;
• caller-declared CPU/memory/GPU reservations;
• exclusive GPU assignment through CUDA_VISIBLE_DEVICES;
• subprocess execution, cancellation, walltime limit;
• content-addressed immutable file/directory artifacts;
• durable events and reconnectable long polling;
• retry generation/fencing;
• stdio and Streamable HTTP MCP transports.

Not yet hardened:

• OS cgroup/job-object hard limits for CPU, memory, and scratch;
• surviving a machine/kernel crash while reattaching already-running children (v0.1 safely marks an interrupted attempt failed and requires explicit retry);
• authentication for non-local HTTP exposure;
• MCP Tasks extension mapping;
• artifact garbage collection and retention policy;
• remote workers or distributed scheduling (not currently a goal).

These omissions are deliberate: v0.1 establishes the minimal kernel boundary before adding optional mechanisms.

Development

python -m venv .venv
. .venv/bin/activate
pip install -e '.[dev]'
pytest -q
ruff check .

Architecture rule

If a feature can be moved outside LinkedRun without breaking generic task persistence, dependency scheduling, process lifecycle, artifact commit, or event durability, it should stay outside LinkedRun. In particular, model/protocol validation, resource prediction, experiment semantics, metric semantics, and formal result publication are external concerns.