mlserve

Self-hosted, low-latency ML inference endpoints for a spare GPU machine. One Python process exposes state-of-the-art open models through two authenticated surfaces that share the same warm model pool:

• a REST API (/v1/embeddings, /v1/summarize, model management)
• an MCP server (/mcp, streamable HTTP) for agentic applications

Designed for small-GPU hardware (tested target: 6 GB VRAM / 16 GB RAM) on Windows, macOS and Linux.

Features

• Embeddings via sentence-transformers (MiniLM, BGE, ...) on CUDA/MPS/CPU
• Summarization via Ollama-served quantized LLMs (llama3.2, gemma3, deepseek-r1, ...) with prompt-guided instructions
• Never dormant — models preload at startup, a keep-warm loop pings idle models, and Ollama runs with keep_alive: -1 so weights stay in VRAM
• Async & parallel — non-blocking endpoints with per-model concurrency limits sized for small GPUs
• Authenticated — API-key middleware (X-API-Key or Bearer) covering REST and MCP
• Observable — structured JSON logs with per-request trace ids, optional Langfuse tracing (free tier)
• Config-driven — add/remove models by editing config/models.yaml; add new runtimes by registering a backend class

Architecture

                ┌─────────────────────────────────────────────┐
   REST clients │  FastAPI app (single process, single worker)│
  ──────────────┤                                             │
   X-API-Key    │  RequestContext ─ ApiKey middleware         │
                │        │                                    │
   MCP clients  │   ┌────┴─────┐        ┌──────────────────┐  │
  ──────────────┤   │ /v1/*    │        │ /mcp (FastMCP)   │  │
   (agents)     │   └────┬─────┘        └───────┬──────────┘  │
                │        └───────┬──────────────┘             │
                │                ▼                            │
                │   EmbeddingService · SummarizationService   │
                │        (logging · tracing · limits)         │
                │                ▼                            │
                │        ModelRegistry (+ keep-warm loop)     │
                │      semaphores · lifecycle · lazy loads    │
                │        ▼                      ▼             │
                │  sentence-transformers      Ollama          │
                │  (CUDA / MPS / CPU)     (quantized LLMs)    │
                └─────────────────────────────────────────────┘

Quickstart

git clone <this-repo> && cd ds-ml-mcp-service
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[embeddings]"

# Ollama powers summarization: https://ollama.com/download
ollama pull llama3.2:3b

cp .env.example .env      # set MLSERVE_API_KEYS to a long random value
mlserve                   # serves on http://127.0.0.1:8000

Full platform guides (Windows/CUDA specifics, Apple Silicon, remote access): docs/SETUP.md.

REST API

export KEY="<your api key>"

# Embeddings — single text or batch
curl -s http://127.0.0.1:8000/v1/embeddings \
  -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  -d '{"input": ["first text", "second text"]}'

# Summarization — instruction-guided
curl -s http://127.0.0.1:8000/v1/summarize \
  -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  -d '{"text": "<long text>", "instruction": "Three bullet points.", "max_words": 80}'

# Ops
curl -s http://127.0.0.1:8000/health                      # unauthenticated
curl -s -H "X-API-Key: $KEY" http://127.0.0.1:8000/v1/models
curl -s -X POST -H "X-API-Key: $KEY" http://127.0.0.1:8000/v1/models/embed-bge-small/load

Interactive OpenAPI docs at http://127.0.0.1:8000/docs.

Every response carries a request_id (also in the X-Request-ID header); grep it in logs/mlserve.jsonl to see the full trace of that call.

MCP

The MCP endpoint lives at http://<host>:8000/mcp (streamable HTTP) and exposes three tools: list_models, embed_texts, summarize_text.

Client configuration (any MCP client that supports HTTP servers + headers):

{
  "mcpServers": {
    "mlserve": {
      "type": "http",
      "url": "http://127.0.0.1:8000/mcp",
      "headers": { "X-API-Key": "<your api key>" }
    }
  }
}

Configuration

Process settings come from env vars / .env (see .env.example); the model catalog lives in config/models.yaml. Highlights:

Setting	Default	Purpose
MLSERVE_API_KEYS	— (required)	comma-separated accepted API keys
MLSERVE_HOST / MLSERVE_PORT	127.0.0.1 / 8000	bind address
MLSERVE_KEEPALIVE_INTERVAL_SECONDS	240	idle-model warm-ping cadence
MLSERVE_LOG_PAYLOADS	true	log truncated input/output previews
MLSERVE_MAX_BATCH_SIZE	64	embedding batch cap
MLSERVE_CORS_ORIGINS	off	browser origins allowed to call the API
LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY	off	enable Langfuse tracing

Adding / removing models

Edit config/models.yaml and restart — or load/unload at runtime via POST /v1/models/{name}/load|unload. New runtimes (llama.cpp, vLLM, remote APIs) are one subclass away; see docs/ARCHITECTURE.md.

Tests

pip install -e ".[dev]"
pytest                    # fast suite, fake backends, no downloads
pytest -m integration     # real-model tests (downloads MiniLM, ~90 MB)

Latest local run: docs/TEST_REPORT.md.

Deployment

Docker + Google Cloud (and generic VM) instructions: docs/DEPLOYMENT.md.

Docs

• docs/SETUP.md — install on Windows (GPU laptop), macOS, Linux
• docs/ARCHITECTURE.md — components, request flow, extension points
• docs/DEPLOYMENT.md — Docker, docker-compose, GCP
• docs/TEST_REPORT.md — latest verified test run