48 lines
2.7 KiB
Markdown
48 lines
2.7 KiB
Markdown
---
|
|
name: backtest-integrity
|
|
description: "Audit any trading backtest for overfit before you trust it. Runs the Lopez-de-Prado integrity battery: purged & embargoed combinatorial-CV (CPCV), Probability of Backtest Overfitting (PBO), Deflated Sharpe Ratio (DSR) with the FULL trial count, sample-uniqueness down-weighting, and triple-barrier labels. Turns 'it backtested great' into a calibrated honesty verdict. By TradeHouse (Bloomberg-terminal-of-prediction-markets)."
|
|
license: MIT
|
|
homepage: https://th-validator.eliteaiempire.com
|
|
tested_with: claude-code v2.x
|
|
---
|
|
|
|
# Backtest-Integrity
|
|
|
|
A great-looking backtest is the default outcome of searching enough strategies — not
|
|
evidence of edge. This skill applies the **Lopez-de-Prado** integrity battery so you only
|
|
trust a result that survives it.
|
|
|
|
## When to use
|
|
- Before promoting a trading strategy / signal / persona to live capital.
|
|
- When a backtest Sharpe looks "too good" — quantify the overfit probability.
|
|
- Reviewing someone else's backtest claim.
|
|
|
|
## The battery
|
|
1. **Purged + embargoed CV / CPCV** — remove train/test leakage from overlapping labels; an
|
|
un-purged CV inflates Sharpe by leaking future info across the split boundary.
|
|
2. **PBO (Probability of Backtest Overfitting)** — across combinatorial train/test splits,
|
|
how often does the in-sample-best config underperform out-of-sample? PBO > 0.5 = the
|
|
selection is no better than noise.
|
|
3. **Deflated Sharpe Ratio (DSR)** — deflate the observed Sharpe by the number of trials
|
|
ACTUALLY searched (the whole grid, not the survivors). Undercounting trials is the #1
|
|
way DSR lies — count the full search space.
|
|
4. **Sample-uniqueness weighting** — concurrent/overlapping labels are not independent
|
|
observations; down-weight them or N is fiction.
|
|
5. **Triple-barrier labels** — label by which of {profit-take, stop, time} is hit first,
|
|
not fixed-horizon returns.
|
|
|
|
## How to run
|
|
1. Take the strategy's returns/labels + the SIZE of the search that produced it (number of
|
|
configs/params/personas tried — be honest, include the discarded ones).
|
|
2. Run purged-CPCV → PBO. Run DSR with trials = full search space. Apply uniqueness weights.
|
|
3. Emit the verdict: `SUPPORTED` (survives) / `OVERFIT_RISK` (PBO high or DSR ≤ 0) with the
|
|
numbers and what would change the verdict.
|
|
|
|
## Output
|
|
`{ PBO, DSR, deflated_sharpe, trials_counted, uniqueness_factor, verdict }` + a plain-English
|
|
"would I bet real money on this?" line.
|
|
|
|
## Upgrade
|
|
The hosted **Overfit Validator** (free lead-magnet) + Research (hypothesis→p-value) + Copilot
|
|
are at **https://th-validator.eliteaiempire.com**, backed by 16-venue prediction-market
|
|
settlement + L2 depth. This skill is the portable integrity logic.
|