X Archive Daemon

Kisa aciklama: X uzerinden post toplayan, bunlari yerel SQLite arsivine yazan ve MCP ile ajanlara kullandiran daemon tabanli bir arsivleme ve akilli arama projesi.

X Archive Daemon archives posts from X into a local SQLite database, exposes them through a daemon-first architecture, and optionally adds a local semantic analysis layer for smarter retrieval.

Install First

If you work with coding agents, the easiest setup flow is:

1. Give the repository link to the agent.
2. Ask the agent to install dependencies.
3. Ask the agent to create the local secret file.
4. Ask the agent to start the daemon and the MCP bridge.

Manual setup:

npm install

Create .secrets/x.json:

{
  "authMode": "bearer_token",
  "bearerToken": "YOUR_X_BEARER_TOKEN"
}

Start the daemon:

npm run daemon:start

Start the MCP bridge:

npm run mcp:start

What This Project Does

This project has three layers:

• ingest
  • fetch posts from X
  • store them in SQLite
• analysis
  • optional
  • adds local embeddings, topic labels, and educational scoring
• semantic search
  • searches analyzed posts locally
  • gives the agent a small, relevant candidate set

Scenario:

• ingest = put boxes into storage
• analysis = attach labels to the boxes
• semantic search = find the right boxes quickly

Architecture

• daemon
  • the real execution engine
  • exposes:
    • GET /health
    • GET /tools
    • POST /invoke
• MCP
  • thin stdio bridge for agents
  • exposes the same tools to the model
• SQLite
  • stores posts, scopes, sync runs, billing estimates, and optional analysis results

Core Features

1. Smart, low-cost ingest

The system avoids paying twice for the same coverage.

Example:

• first you fetch the latest 50
• later you ask for the latest 100
• it does not re-fetch the first 50
• it fetches only the missing 50

This works for:

• latest N timeline
• latest N original posts
• exact repeated date windows
• exact repeated search queries

2. Original posts by default

When a user generically says "fetch posts" or "archive tweets", the default tool is:

• ingest.accounts.original_backfill

This excludes:

• replies
• retweets
• quote tweets

That keeps the archive cleaner and cheaper.

3. Media URLs are stored, files are not downloaded

If a post contains images:

• files are not downloaded
• only mediaUrls are stored

This keeps disk and network usage low.

4. Optional analysis layer

Analysis is off by default.

That means:

• you can archive posts without analyzing them
• you can analyze the same archive later
• old and newly ingested posts are both supported

5. Local semantic search

Once analysis exists, you can search with natural language prompts like:

• "teaching posts about coding"
• "monolith vs microservices"
• "backend and architecture advice"

The system then:

• searches the analyzed local archive
• scores candidates locally
• lets the agent work on a narrow, relevant subset

Tool Groups

• sources.accounts.resolve
• ingest.accounts.backfill
• ingest.accounts.original_backfill
• ingest.accounts.sync
• ingest.search.backfill
• archive.posts.list
• archive.posts.search
• archive.posts.semantic_search
• archive.accounts.list
• archive.accounts.get
• archive.billing.summary
• analysis.posts.run
• analysis.labels.list
• archive.insights.summary

MCP also exposes:

• system.daemon.start

Risk Model

safe-read

• read-only
• does not change X or the local archive

operator-write

• writes into the local SQLite archive
• may consume X API credits
• does not create, delete, like, reply, or DM on X

Note:

• analysis.posts.run is also operator-write
• but it does not consume X API credits
• it only writes local analysis data and uses CPU

Performance Reference

Measured system:

• Apple Silicon M4 mini
• 16 GB RAM
• local embedding model on CPU

Measured analysis speed:

• 900 posts = about 30.69s
• 100 posts = about 3.41s
• 980 posts = about 33.42s expected total

Important:

• this is not a large generative LLM benchmark
• this is local tagging + embedding + semantic retrieval preparation

Local Model

The current analysis layer uses one small local model:

• sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

Its role is:

• post embeddings
• label-description embeddings
• query embeddings
• similarity-based tagging
• semantic retrieval

This layer does not generate final answers. It builds a local meaning layer on top of the archive.

How Analysis Works

When analysis runs, each post gets signals such as:

• educational score
• matched labels
• topic scores
• reply/noise/technical flags

Tagging is based on:

• rule-based signals
• embedding similarity
• a fixed label catalog with descriptions

Label Catalog

The repository includes a versioned label catalog with Turkish descriptions.

Examples:

• software_architecture
• monolith_vs_microservices
• backend_api
• database_sql
• database_indexing
• caching
• distributed_systems
• testing_qa
• clean_code
• code_review
• security_appsec
• authentication_authorization
• ci_cd_release
• ai_assisted_coding
• vibe_coding
• prompting_for_engineering
• technical_decision_making

Quick Start

Check daemon health:

curl http://127.0.0.1:3200/health

List available tools:

curl http://127.0.0.1:3200/tools

Estimate original-post ingest:

curl -X POST http://127.0.0.1:3200/invoke \
  -H 'content-type: application/json' \
  -d '{
    "tool": "ingest.accounts.original_backfill",
    "input": {
      "username": "sampleauthor",
      "searchMode": "recent",
      "targetCount": 100,
      "estimateOnly": true
    }
  }'

Run local analysis:

curl -X POST http://127.0.0.1:3200/invoke \
  -H 'content-type: application/json' \
  -d '{
    "tool": "analysis.posts.run",
    "input": {
      "username": "sampleauthor",
      "limit": 200,
      "onlyUnanalyzed": true
    }
  }'

Run semantic search:

curl -X POST http://127.0.0.1:3200/invoke \
  -H 'content-type: application/json' \
  -d '{
    "tool": "archive.posts.semantic_search",
    "input": {
      "username": "sampleauthor",
      "query": "teaching posts about coding",
      "educationalOnly": true,
      "limit": 10
    }
  }'

Model Packaging Note

The repository is public and safe to clone, but the local model files are not stored in git history because of GitHub file size limits.

The codebase is public-ready. Model packaging is handled separately from normal git push history.