Backtest Overfitting: When the Best Strategy Is Just the Luckiest

A spectacular equity curve can be the product of a weak strategy and an extensive search. If a researcher tries many moving-average windows, filters and entry rules, then publishes only the winner, the relevant question is not simply whether that winner made money. It is whether the entire selection procedure can identify an edge on data it has not already used.

A backtest is evidence about a process, not a certificate attached to a curve. Before treating its result as actionable, establish how many alternatives were considered, when each decision was made, what information was available at execution, and what remains after realistic costs. A high Sharpe ratio cannot answer those questions by itself.

Why 100 attempts are not one experiment

Consider a deliberately simple null model: every candidate has no genuine edge; each statistical test has a 5% false-positive probability; and the tests are independent. The chance of avoiding a false positive in all m tests is 0.95 raised to m. Therefore:

P(at least one false positive) = 1 − (1 − 0.05)^m

With 100 independent tests, at least one apparently significant result is almost certain under these assumptions. That does not mean any particular winning strategy has a 99.41% probability of being false, nor that 99.41% of strategies lose money. The calculation describes the chance of at least one false alarm across an all-null experiment. It is not a posterior probability and not a calculation of the academic probability-of-backtest-overfitting metric.

Real trading variants usually share signals and returns, so they are not independent. Do not insert a raw count of correlated variants into this formula and label the result an exact estimate. The illustration establishes why search breadth matters; it does not measure the effective number of independent experiments in your database.

Count the search that produced the winner

The experiment log must include discarded ideas, not just saved model files. Trying ten windows on five markets already creates more choices than a report showing one window on one market suggests. Changing the start date, excluding an awkward crisis, reversing a signal, adding a volatility filter or selecting the prettiest performance metric can all use information from the same history.

Record the data version, candidate rule, parameters, cost assumptions, evaluation dates and result for every run. Preserve the order of decisions. A story about why a signal ought to work is useful, but a story invented after viewing its profits is not independent evidence.

For a fixed, fully disclosed family of m valid tests, a simple conservative control is to test each at α/m. With 20 tests and a 5% family-wide error budget, the threshold is 0.0025: an unadjusted p-value of 0.01 does not clear it. This Bonferroni rule does not require independence. It does require valid individual tests and an honestly specified test family. It cannot repair look-ahead bias, missing trials or a poorly modelled return distribution.

A holdout stops being a holdout when it guides revisions

Keep three functions separate: fitting parameters, choosing among candidate designs, and finally evaluating a frozen research process. Validation data help choose the design; a final untouched holdout evaluates it. Once a disappointing holdout prompts another filter, threshold or market choice, that period has become development data. Calling it out-of-sample again does not restore its independence.

An illustrative rolling design fits observations 1–500 and evaluates 501–600, then fits 101–600 and evaluates 601–700, then fits 201–700 and evaluates 701–800. Observations 801–1000 remain locked for a final assessment. The schedule and selection rule must be fixed before examining the evaluation results. Earlier test observations may enter later training when time has genuinely advanced; this is not permission to choose a new policy after inspecting every test fold.

Combine nonoverlapping test-period returns in chronological order and include boundary trades and costs. If labels use future returns or positions overlap, remove training observations whose information reaches into the evaluation interval. Any gap must follow the actual information horizon, not a ritual fixed number of days. Randomly shuffling rows can leak tomorrow's market into yesterday's training.

Data construction is a separate hurdle. In futures, the choice of continuous price series and roll accounting can change both signals and reported profits before model selection even starts.

A small cost error can consume the remaining edge

Suppose 200 completed, equal-notional trades earned an average gross 8 basis points per round trip. At a round-trip cost of 6 basis points, the arithmetic average net result is 2 basis points per trade. At 10 basis points, it is −2. These are per-trade averages, not annualized or compounded portfolio returns; overlapping positions and capital usage still need their own accounting.

Include both sides of the spread, commissions, slippage and applicable financing or borrow costs. Do not subtract a one-way cost from a round-trip return. Test costs where turnover is greatest and liquidity is weakest, rather than assuming one calm-day spread applies everywhere. A statistically unusual gross return may still be commercially unusable.

Then ask whether modest parameter changes destroy the result, whether one market episode supplies most profits, and whether a few correlated trades inflate the apparent sample size. Stability across nearby settings strengthens the case but does not replace untouched data. Neither a universal minimum trade count nor a universal Sharpe cutoff can resolve dependence, tail risk and selection bias.

Audit the decision before trusting the curve

1. Reproduce the inputs. Establish point-in-time availability, executable prices, corporate actions or contract rolls, and a versioned dataset.
2. Reconstruct the search. Recover failed trials and every discretionary exclusion; missing search history means the strength of selected evidence is unknown.
3. Freeze the protocol. Specify fitting windows, candidate selection, position handling and a cost model before evaluation; reserve genuinely unseen data.
4. Recalculate the net result. Reconcile individual trades with portfolio returns, stress execution costs and examine concentration and dependence.
5. Define what would overturn the finding. A result that vanishes after leakage removal, realistic costs or locked evaluation fails its claimed test. An uncertain result remains a research candidate, not a demonstrated edge.

The most informative backtest is not necessarily the one with the smoothest line. It is the one whose data, search history and evaluation rules make it possible for somebody else to discover that the conclusion is wrong.