filings-search

Hybrid (BM25 + kNN) agentic retrieval over SEC 10-K filings, on OpenSearch, with a Claude tool-use agent that plans its own searches and answers with per-claim citations and a hard numeric-grounding check. FastAPI surface + TypeScript MCP tool server. Built as the OpenSearch successor to my Qdrant pipeline; the cite-or-abstain grounding is ported from my 12-domain provenance wrapper.

EDGAR ──► connector ──► section splitter ──► chunker ──► embeddings ──► OpenSearch (BM25 + HNSW kNN)
(SEC API)  (ticker→CIK,   (Item 1/1A/1C/7/8…)  (450 tok,    (nomic-embed-text            │
            10-K list,     order-constrained,    60 overlap)  local via Ollama, or        │
            iXBRL strip)   x-ref filtered)                     OpenAI)                     ▼
                                                                          hybrid search (RRF) + filters
                                                                                       │
                                                       ┌───────────────────────────────┼─────────────────────┐
                                                       ▼                               ▼                     ▼
                                              FastAPI /search /ask           TS MCP server            eval harness
                                              /chunk /companies         (search_filings, ask_filings)  (P@k/MRR, LLM-judge)
                                                       ▲
                                              Claude agent (tool loop):
                                              resolve_company → search_filings (item/ticker/FY filters, hybrid|bm25|knn)
                                              → expand_chunk → answer with [c:chunk_id] cites → numeric grounding check

What's in it

Layer	File	Notes
Connector	filings_search/edgar.py	SEC ticker map + browse-edgar fallback (handles successor-shell CIKs), data.sec.gov submissions, primary-doc fetch, iXBRL/HTML strip, disk cache
Parsing	filings_search/parse.py	10-K Item splitter: title-verified headings, cross-reference filter, canonical-order + longest-span selection (defeats ToC rows)
Chunking	filings_search/chunk.py	paragraph-respecting, token-bounded (450/60 overlap), deterministic chunk ids `sha1(accession
Embeddings	filings_search/embed.py	nomic-embed-text (768-d) on local Ollama by default; OpenAI text-embedding-3-small fallback
Index	filings_search/index.py	OpenSearch 2.19 mapping: english analyzer BM25 field + knn_vector (lucene HNSW, cosine) + keyword metadata (ticker, cik, item, fiscal_year, accession…)
Retrieval	filings_search/search.py	bm25, knn, hybrid (client-side reciprocal-rank fusion), metadata filters, neighbor expansion
Agent	filings_search/agent.py	Claude (claude-opus-5) tool loop; tools: list_indexed_companies, resolve_company, search_filings, expand_chunk; answer must cite [c:id]; every figure in the answer must appear in a cited chunk or the response is flagged grounded_numbers=false
API	filings_search/api.py	FastAPI: GET /search, POST /ask, GET /chunk/{id}?expand=, GET /companies, GET /health
MCP	mcp/src/server.ts	TypeScript stdio MCP server exposing list_companies, search_filings, get_chunk, ask_filings
Eval	eval/	run_retrieval_eval.py (hit@1/hit@5/MRR@10 by mode, labeled queries), run_grounding_eval.py (numeric grounding + LLM-as-judge citation support)

Run it

docker compose up -d                       # OpenSearch 2.19 (knn + neural plugins), :9200
ollama pull nomic-embed-text               # local embeddings
python3.12 -m venv .venv && ./.venv/bin/pip install -r requirements.txt
./.venv/bin/python ingest.py --recreate AAPL MSFT NVDA JPM XOM WMT TSLA JNJ   # ~2 min, 8 filings, ~2.1k chunks
./run_api.sh                               # FastAPI on :8801
./.venv/bin/python -m filings_search.agent "What does NVIDIA disclose about export controls to China?"
./.venv/bin/python eval/run_retrieval_eval.py
./.venv/bin/python eval/run_grounding_eval.py

MCP (Claude Desktop / Claude Code / Cursor):

{"mcpServers": {"filings-search": {"command": "node", "args": ["/ABS/PATH/filings-search/mcp/dist/server.js"],
                                    "env": {"FILINGS_API_URL": "http://127.0.0.1:8801"}}}}

Config via env: FS_OPENSEARCH_URL, FS_INDEX, FS_EMBED_BACKEND=ollama|openai, FS_AGENT_MODEL, FS_JUDGE_MODEL, SEC_USER_AGENT. ANTHROPIC_API_KEY (or ~/.env) for the agent/judge.

Results (2026-08-18, 8 filings / 2,101 chunks, 28 labeled queries)

Retrieval — see eval/retrieval_results.json:

setting	mode	hit@1	hit@5	MRR@10	avg ms
unfiltered	bm25	0.571	0.857	0.686	5.7
unfiltered	knn	0.679	0.857	0.759	29.7
unfiltered	hybrid	0.679	0.857	0.759	37.9
ticker_filtered	bm25	0.679	0.929	0.772	3.8
ticker_filtered	knn	0.679	0.893	0.779	26.6
ticker_filtered	hybrid	0.714	0.929	0.812	34.9

Agent grounding — 10 analyst questions, claude-opus-5 agent + claude-opus-5 judge (see eval/grounding_results.json):

metric	value
numeric-grounded rate (every figure appears in a cited chunk)	10/10 = 1.00
LLM-judge verdict: grounded / partially grounded / ungrounded	9 / 1 / 0
mean fraction of claims supported by a cited chunk	0.968
avg citations per answer	9.7
avg tool calls per answer (agent-chosen searches/expansions)	7.6
avg latency	41 s (13 s simple → 59 s multi-Item)
tokens for the 10-question run	563k in / 23k out

The one partially_grounded (Walmart tariffs, 0.87) was the agent summarizing a mitigation that the cited chunk states more narrowly — the judge caught it; that is what the judge is for.

Design notes / honest limitations

• Agentic ≠ fixed RAG. The model chooses company, Item, mode, and how many rounds; a fixed top-k pipeline gets no second chance. The tool descriptions carry the "when to use" guidance (item map, bm25-for-figures, widen-if-empty).
• Grounding is strict on purpose. A derived rounding ("$99,779M" → "~$99.8B") is flagged as ungrounded; analysts want the figure as filed. Loosen with a tolerance if you disagree.
• 10-K structure quirks are real, not parser bugs: JPM and XOM are "wrapper" 10-Ks whose MD&A/financials sit in a back-of-book Financial Section (labeled under the last Item); NVIDIA files statements under Item 15. Eval labels for those are ticker-only. A follow-up is F-page detection (Consolidated Statements of … headings) to relabel as Item 8.
• RRF is client-side — transparent and easy to reason about; OpenSearch's hybrid query
  • normalization pipeline is the in-cluster alternative. No cross-encoder re-ranker yet.
• Single filing per company in this run; --filings N pulls prior years (fiscal_year filter already in the mapping and tools).
• No auth, no rate limits, single-node OpenSearch — this is a working vertical, not a deployment.