backtest-reality-check

An MCP server that tells you whether your backtest survives its own statistics.

Point any MCP client at it and ask "is my backtest real?". It answers with arithmetic — deflated Sharpe, minimum track record length, multiple-testing correction, fill realism — using the checks professional quant research applies and hobby trading bots almost never do.

No market data. No API keys. No broker connection. It cannot recommend a trade and has nothing to sell. It can only ever tell you your number is weaker than you think, or that you can't tell yet. Those are the only two honest answers available.

▶ 60-second demo — the simulation, the maths, and the tool failing on my own dead bot.

---

Why it exists

I built a sports-betting engine. The backtest reported a Sharpe ratio of 8.61. It went 0 for 38 live.

Two things were wrong, and only one of them is the famous one:

1. Overfitting. I'd tried ~25 variants and kept the best. Nobody counts that.
2. Broken data. The P&L was mis-computed and the fills were at prices the market never offered — 25.5% of them better than anything actually available.

This tool catches both, with two different detectors, because they are different failures and a Sharpe ratio can only see one of them.

The honest limitation, stated up front

Deflation catches overfitting and luck. It cannot detect broken data.

Run my own dead bot through check_backtest — Sharpe 8.61, 0.3 years, 25 trials — and it returns SURVIVES at 99.4%. Because a Sharpe that high genuinely is extraordinary if the returns are real. Mine weren't.

That is why check_fills exists. A backtest must pass both.

The five tools

Tool	Question it answers
check_backtest	Does your Sharpe survive multiple-testing correction?
noise_sharpe	What does the best of N edgeless strategies score over this sample?
how_long_to_prove	How many years before this Sharpe is distinguishable from zero?
sample_size_check	Can your sample even detect the edge you claim?
check_fills	Are your fills earnable, or better than the market ever offered?

The input people omit is the trial count — how many variants you tried before settling on this one. Every threshold, lookback and filter you swapped counts. It is the number that decides the answer, and almost nobody reports it.

Install

"mcpServers": {
  "backtest-reality-check": {
    "command": "python3",
    "args": ["/absolute/path/to/backtest-reality-check/src/server.py"]
  }
}

Pure standard library. No dependencies, no build step, Python 3.8+.

Try it

> check_backtest --sharpe 2.13 --years 1 --trials 34
noise alone at 34 trials  -> Sharpe 2.13
>>> FAILS — your Sharpe is at or below what pure noise produces

Thirty-four variants over one year makes a Sharpe of 2.13 the expected result of strategies with no edge whatsoever. That is the whole idea.

Some reference points from how_long_to_prove:

Sharpe	Years of daily data before it's distinguishable from zero
1.5	1.2
1.0	2.7
0.7	5.5
0.5	10.8
0.35	22

Verification

The maths is checked against published values, and these are hardcoded as tests:

• MinTRL at SR 1.0 → 2.71 years
• Max trials for SR 1.0 over 5 years → 45
• Over 2 years → 7
• E[max Sharpe] at N=45, 5 years → 1.00

python3 tests/test_anchors.py

A bug I shipped and caught with exactly these: expected_max_sharpe first returned the raw order-statistic term with no sample-length scaling, making the bar independent of backtest length so short backtests wrongly passed. Under the null the per-observation Sharpe has SD ≈ 1/√T, so the annualised benchmark is max_z / √years. My own tool failed its own sanity check.

Sources

Bailey & López de Prado, The Deflated Sharpe Ratio (JPM 2014) and The Sharpe Ratio Efficient Frontier (J. Risk 2012); Bailey, Borwein, López de Prado & Zhu, Pseudo-Mathematics and Financial Charlatanism (Notices of the AMS, 2014); Harvey, Liu & Zhu (RFS 2016) for the multiple-testing hurdle.

Licence

MIT. Use it, fork it, and please do run it on mine.