explain-lint

A linter for unexplained terms in prose.

Compilers catch "undefined variable." Prose has the same bug: a term used without ever being explained. AI-generated text is especially prone to it — it drops jargon fluently and never defines it. explain-lint is the prose analogue of a linter. It finds every term's first occurrence, remembers it in a ledger, and — differentially — surfaces only what changed since last time, so the expensive "is this actually explained?" judgment (by a human or an LLM) runs on the diff, not on the whole document every time.

Origin: spun out of a 23-chapter physics paper where an AI reviewer noticed the term "P&H" was used ~40 times with no definition. The author's reaction — "you can detect an undefined variable even in your own writing" — became this tool.

Why differential matters

Re-reading a whole book to ask "is every term explained?" is expensive whether a human or an LLM does it. explain-lint keeps a ledger of each term's first appearance, keyed by a hash of that line's text — not its line number. So:

• Add a paragraph up front and every line number shifts? No noise. The hashes still match; --sync rewrites the numbers silently.
• Actually reword where a term first appears? That one term is flagged MOVED — re-judge it, nothing else.
• A genuinely new term appears? It's flagged NEW — judge it once, and it's remembered forever.

The cost of the human/LLM judgment collapses from O(document) every run to O(what you changed).

Install

Pure Python 3, no dependencies.

git clone <this repo>
python -m explain_lint your_doc.md

Quick start

# 1. See every term's first occurrence (seed material for a ledger)
python -m explain_lint doc.md --dump

# 2. Build doc.md.terms.md (the ledger) — judge each term's category/explained
#    (by hand, or hand the NEW list to an LLM). See format below.

# 3. From now on, this is all you run. It stays silent until something changes:
python -m explain_lint doc.md          # exit 0 = nothing new; exit 1 = NEW/MOVED

# After edits that only shift line numbers:
python -m explain_lint doc.md --sync   # rewrite line numbers for unchanged terms

Multiple files are read in order (useful for a book split into chapters):

python -m explain_lint ch01.md ch02.md ch03.md --ledger book.terms.md

See examples/sample.md (English) and examples/sample_ja.md (Japanese) for working examples with pre-built ledgers. PDF samples are at examples/sample.pdf and examples/sample_ja.pdf (requires: pip install explain-lint[pdf]).

The ledger

A plain Markdown table you (or an LLM) edit. The tool maintains the first four columns mechanically; you own explained and notes.

| term | category | first_seen | hash | explained | notes |
|---|---|---|---|---|---|
| ホロノミー | needs-explanation | ch12.md:42 §Geometry | ab12cd34 | no  | GAP: used with no gloss |
| Markov     | needs-explanation | ch01.md:5  §Intro    | 50632610 | yes | glossed inline on first use |
| Introduction | exclude         | ch01.md:3  §Intro    | d69dd448 | na  | heading word |

Suggested vocabularies (not enforced — they are just strings):

• category: needs-explanation / common / proper-noun / exclude
• explained: yes / no / na

The actionable output is explained = no — terms that need a gloss and don't have one. Grep them straight out of the ledger:

grep '| no |' doc.md.terms.md

What it does — and doesn't

Does (the machine-detection core):

• Extract candidate terms: katakana runs (--min-kana, default 3) and/or Latin words and abbreviations like AT&T / R&D / Peacock-Hall (--min-latin, default 3). Toggle with --no-kana / --no-latin.
• Record each term's first occurrence: file, line, nearest heading, line-hash.
• Ignore fenced code, inline code, $math$, $$math$$, image tags, and URLs.
• Diff against the ledger → NEW / MOVED / GONE; exit 1 on NEW or MOVED.
• PDF input support: pypdf extracts text per page; page numbers replace line numbers in first_seen (e.g. book.pdf:p42 §Geometry). Requires: pip install explain-lint[pdf].
• Back-of-book index: --index generates a sorted term → page/line listing from the ledger.
• Morphological analysis: --morph extracts kanji/hiragana nouns via Janome. Requires: pip install explain-lint[ja].

Does not (on purpose):

• Decide whether a term is explained. That verdict is a separate human/LLM layer written into the ledger. The tool only keeps the ledger honest against the text and tells you what is new to judge. This keeps the core deterministic, offline, dependency-free, and auditable; the (fuzzy, model-dependent) judgment stays a pluggable step you control.

Use it from an AI assistant (MCP)

The judgment step is an AI task — but a cheap one, because the differential core hands the assistant only the NEW/MOVED terms, not the whole document. The included MCP server (explain_lint_mcp.py) is the seam: connect an assistant (Claude, etc.) and it can run the whole loop itself.

pip install mcp        # the core needs nothing; the server needs this

Register it (Claude Code / Claude Desktop mcpServers config):

{
  "mcpServers": {
    "explain-lint": {
      "command": "python",
      "args": ["/absolute/path/to/explain-lint/explain_lint_mcp.py"]
    }
  }
}

Tools the assistant sees:

tool	purpose
lint_report(paths, ledger)	the diff: NEW / MOVED / GONE terms
get_term_context(term, paths, window)	first occurrence + surrounding lines, to judge
record_judgment(ledger, term, category, explained, notes, paths)	write the verdict back
list_gaps(ledger)	the finding: terms marked explained = no
sync_ledger(paths, ledger)	refresh line numbers after pure drift
dump_terms(paths)	every first occurrence (seed a ledger)

The loop the assistant runs: lint_report → for each NEW term get_term_context → judge → record_judgment → list_gaps to report the unexplained terms to you. The server holds no model and makes no API calls — the intelligence is whatever client connects to it.

Roadmap

Done: the deterministic core (CLI + importable functions) and the MCP server that lets an assistant drive the judgment loop. Envisioned next:

• A one-call convenience tool that runs the whole loop server-side when the client passes it an LLM callback.
• Richer term extraction (multi-word phrases; language packs beyond kana+Latin).
• A CI action (fail a PR when a new unexplained term lands in docs).
• An editor integration (flag the gap as you write).

Development

The core has no runtime dependencies; the tests need only pytest:

pip install pytest
python -m pytest

The suite (tests/) covers the differential lifecycle (NEW/MOVED/GONE and that a re-judged MOVED clears), ledger round-tripping through special characters, term extraction and exclusions, sync line-drift vs content change, and that the CLI survives a non-UTF-8 console. Each is also a regression test for a specific past bug — see issues/.

License

MIT — see LICENSE.