/backtesting

name: backtesting
description: Use when testing a trading strategy on historical data. Covers the biases that make a backtest lie, realistic costs, walk-forward validation, and the statistics that distinguish an edge from a coincidence. Trigger on "backtest this strategy", "my backtest looks too good to be true", "why does live trading underperform my backtest". Do not use for testing whether a single signal is statistically significant — see the quantitative-analysis skill.
metadata:
  category: finance
  version: 1.0.0
  tags: [backtesting, validation, overfitting, bias, statistics]

Backtesting

Purpose

Test a strategy honestly. The default outcome of a backtest is a beautiful equity curve that describes the past and predicts nothing — and every bias in the process pushes in that direction.

When to Use

• Testing a strategy before risking capital.
• Validating that an apparent edge is real.
• Diagnosing why a live strategy underperforms its backtest.
• Reviewing someone else's backtest.

Capabilities

• Bias elimination: survivorship, look-ahead, selection.
• Realistic cost modeling: commissions, spread, slippage, borrow.
• Walk-forward and out-of-sample validation.
• Statistical significance and multiple-testing correction.
• Robustness testing.

Inputs

• The strategy rules, stated precisely enough to be executed mechanically.
• Point-in-time historical data, including delisted instruments.
• Realistic assumptions about execution.

Outputs

• Performance metrics with costs included.
• Out-of-sample results, reported honestly.
• A judgment on whether the edge is real, with the evidence.

Workflow

1. **Eliminate the biases before you run anything** — Survivorship (delisted names must be in the universe), look-ahead (data must be used only when it was available), and selection (the strategy must be specified before it is tested). 2. **Model the costs honestly** — Commission, the bid-ask spread, slippage that scales with size and inversely with liquidity, and borrow costs on shorts. Most published edges are smaller than realistic costs. 3. **Split the data before you look at it** — In-sample for development, out-of-sample for validation, and a final hold-out you touch exactly once. 4. **Walk forward** — Re-fit on a rolling window and test on the period immediately after. A single in-sample fit tells you nothing about a strategy that will be run forward in time. 5. **Correct for multiple testing** — If you tested forty variants, the best one looks good by chance. A p-value that has not been adjusted for the number of things you tried is not a p-value. 6. **Stress it** — Perturb the parameters. A strategy that only works with a 14-day lookback and fails with 13 or 15 is fitted to noise.

Best Practices

• The most common backtesting error is look-ahead bias, and it is usually accidental: using a closing price to make a decision that would have been made during the day, or using a fundamental figure on the fiscal date rather than the filing date.
• Transaction costs kill more strategies than any other factor. A strategy trading daily with a 0.05% edge per trade is a losing strategy after a 0.10% round-trip cost, and it will have looked excellent before costs.
• A Sharpe ratio above 2 on a simple strategy in a liquid market is a bug, not a discovery. Look for the bias.
• Parameter sensitivity is the honesty test. If performance collapses when a parameter moves by one unit, the strategy has fitted noise. Plot the surface.
• The out-of-sample period must be genuinely untouched. If you look at it, adjust, and look again, it is now in-sample and you have no validation left.
• Report the strategy's worst period, not just its average. The maximum drawdown and its duration determine whether it is survivable in practice.

Examples

**A backtest that does not lie to itself:**

def backtest(strategy: Strategy, start: date, end: date) -> BacktestResult:
    equity, trades = [], []

    for day in trading_days(start, end):
        # Point-in-time universe: includes everything that was tradable THAT DAY,
        # including companies that were later delisted or went to zero.
        universe = universe_as_of(day)

        # Point-in-time fundamentals: as reported, on or before this date. Not
        # as later restated, and never before the filing date.
        data = fundamentals_as_of(day, universe)

        # Signals are computed on data available at the CLOSE of the prior day.
        # Trades execute at the NEXT day's open. Using today's close to decide
        # and today's close to fill is the classic look-ahead error.
        signals = strategy.signals(data.shift(1))

        for signal in signals:
            fill = next_open(signal.symbol, day)
            cost = transaction_cost(
                price=fill,
                shares=signal.shares,
                adv=data.loc[signal.symbol, "adv_20d"],
            )
            trades.append(Trade(day, signal.symbol, fill, cost))

        equity.append(mark_to_market(day))

    return BacktestResult(equity=equity, trades=trades)


def transaction_cost(price: float, shares: int, adv: float) -> float:
    """Commission + half the spread + slippage that scales with participation."""
    notional = price * shares
    commission = max(1.0, shares * 0.005)
    spread_cost = notional * 0.0005                       # half-spread, liquid names

    # Market impact grows with the fraction of daily volume you consume.
    participation = (shares * price) / max(adv, 1)
    impact = notional * 0.10 * math.sqrt(participation)   # square-root impact model

    return commission + spread_cost + impact

**The result, reported honestly:**

Strategy: momentum + quality, 25 names, monthly rebalance.

                        In-sample    Out-of-sample   Difference
                        2010-2018    2019-2024
CAGR (gross)              18.4%          11.2%
CAGR (net of costs)       14.1%           6.8%