agent-skills/skills/backtest-integrity/SKILL.md

name	description	license	homepage	tested_with
backtest-integrity	Audit any trading backtest for overfit before you trust it. Runs the Lopez-de-Prado integrity battery: purged & embargoed combinatorial-CV (CPCV), Probability of Backtest Overfitting (PBO), Deflated Sharpe Ratio (DSR) with the FULL trial count, sample-uniqueness down-weighting, and triple-barrier labels. Turns 'it backtested great' into a calibrated honesty verdict. By TradeHouse (Bloomberg-terminal-of-prediction-markets).	MIT	https://th-validator.eliteaiempire.com	claude-code v2.x

Backtest-Integrity

A great-looking backtest is the default outcome of searching enough strategies — not evidence of edge. This skill applies the Lopez-de-Prado integrity battery so you only trust a result that survives it.

When to use

• Before promoting a trading strategy / signal / persona to live capital.
• When a backtest Sharpe looks "too good" — quantify the overfit probability.
• Reviewing someone else's backtest claim.

The battery

1. Purged + embargoed CV / CPCV — remove train/test leakage from overlapping labels; an un-purged CV inflates Sharpe by leaking future info across the split boundary.
2. PBO (Probability of Backtest Overfitting) — across combinatorial train/test splits, how often does the in-sample-best config underperform out-of-sample? PBO > 0.5 = the selection is no better than noise.
3. Deflated Sharpe Ratio (DSR) — deflate the observed Sharpe by the number of trials ACTUALLY searched (the whole grid, not the survivors). Undercounting trials is the #1 way DSR lies — count the full search space.
4. Sample-uniqueness weighting — concurrent/overlapping labels are not independent observations; down-weight them or N is fiction.
5. Triple-barrier labels — label by which of {profit-take, stop, time} is hit first, not fixed-horizon returns.

How to run

1. Take the strategy's returns/labels + the SIZE of the search that produced it (number of configs/params/personas tried — be honest, include the discarded ones).
2. Run purged-CPCV → PBO. Run DSR with trials = full search space. Apply uniqueness weights.
3. Emit the verdict: SUPPORTED (survives) / OVERFIT_RISK (PBO high or DSR ≤ 0) with the numbers and what would change the verdict.

Output

{ PBO, DSR, deflated_sharpe, trials_counted, uniqueness_factor, verdict } + a plain-English "would I bet real money on this?" line.

Upgrade

The hosted Overfit Validator (free lead-magnet) + Research (hypothesis→p-value) + Copilot are at https://th-validator.eliteaiempire.com, backed by 16-venue prediction-market settlement + L2 depth. This skill is the portable integrity logic.