/backtest-data-prep

name: backtest-data-prep
description: Build a clean, point-in-time, ready-to-backtest OHLCV dataset for a US equity universe across an arbitrary date window. Emits parquet plus a manifest plus an edge-case log, with corporate actions reconciled, survivorship treatment documented, holidays and half-days preserved correctly, and any IPO partial coverage or trading halts flagged. The dataset is the artifact a downstream Python/R/Julia backtester consumes; the rendered summary is the operator-readable companion. First skill in the suite that ships dataset output mode.

backtest-data-prep

You hand over a universe definition, a date window, and an output directory. The skill emits a clean OHLCV+volume parquet plus a manifest plus an edge-case log. The downstream consumer is a backtester (your Python loop, QuantConnect, vectorbt, zipline-reloaded, etc.), not a human reader.

The skill's value is correctness. Quants spend 80% of their time on data prep, and the most common backtest bugs come from five sources:

1. **Survivorship bias.** Backtesting against today's top-500 silently excludes 2022-2024 failures (FRC, SVB, BBBY, SI). The remaining cohort outperforms the true historical population because the losers are gone. 2. **Look-ahead bias on fundamentals.** Using the latest revised consensus instead of point-in-time. The signal contains forward-looking information that wasn't available at the rebalance. 3. **Corporate action mis-adjustment.** Pre-split prices not adjusted, or adjusted using the wrong factor. 4. **Holiday and half-day handling.** Missing days treated as errors or filled wrong. 5. **Inconsistent calendar alignment.** Calendar dates vs trading dates, US vs global market hours.

The skill addresses all five and documents the treatment in the manifest.

When to invoke

• A quant says "I need clean OHLCV for the top 500 from 2022 to today

for my mean-reversion backtest"

• A researcher needs a survivorship-clean dataset for a paper
• A factor-research extension wants a longer window than the in-memory

panel can carry

• The user says "prep a backtest dataset", "build me a clean OHLCV

parquet", "I need point-in-time prices for a universe"

What you need

• A universe (CLI flag: `top100`, `top500`, `top1000`, `sp500`,

`custom:path/to/tickers.csv`)

• A window (`--window YYYY-MM-DD..YYYY-MM-DD`)
• An output directory (`--out path/to/dataset/`)
• `MASSIVE_API_KEY` exported in the environment
• Stocks Starter plan minimum (flat files included with any paid plan;

flat-files entitlement probed and REST fallback used when not granted)

Expected runtime

A 4-year top-100 run:

• Flat-files cold: ~5-10 minutes
• REST grouped cold: ~10-15 minutes
• Warm (parquet day-cache hit): under 60 seconds

A 5-year top-500 run is roughly 5x the REST time and 2-3x the flat-files time (the bottleneck shifts from day-bucket fetch to per-ticker corp action calls).

What you get back

Three files in the output directory, plus a rendered summary printed to stdout.

**Files:**

• `ohlcv.parquet`: one row per ticker x trading day. Columns documented

in [`references/output-formats.md`](./references/output-formats.md).

• `manifest.md`: human-readable run record. Universe definition, window,

survivorship treatment, corp action methodology, source endpoints, every parameter that affects reproducibility, run timestamp.

• `edge-cases.log`: line-delimited JSON. One entry per edge case

detected: trading halt, IPO partial coverage, delisting during window, ticker change, data gap.

**Layer 1: canonical JSON** matching [`output-schema.json`](./output-schema.json). Emitted to stdout alongside the rendered summary so downstream tools that wrap the CLI can parse the run metadata without re-reading the manifest.

**Layer 2: rendered dataset summary** in the style of a clean-data report (think QuantConnect's data quality report or Bloomberg's BDH audit). See [`references/rendering.md`](./references/rendering.md). Files written, universe construction, corporate actions applied, coverage, edge cases, schema, sources, take. The take answers the question a quant asks at the top of every backtest: "is this dataset actually clean enough to trust?"

How it works

1. **Construct the universe** per [`references/survivorship-handling.md`](./references/survivorship-handling.md). For the default top-N seeds, pull `/v3/reference/tickers` (active and delisted both), enrich with market cap from `/v3/reference/tickers/{ticker}`, rank, and keep the top N. The universe label records whether the seed is forward-looking biased (it usually is for current top-N). 2. **Filter to common stock.** Reference-endpoint `type` field; drop ETFs, ETNs, ADRCs, units, warrants, rights. Documented in the manifest because the consumer may want to override. 3. **Pull daily aggregates** for the window. Flat-files preferred (`s3://flatfiles/us_stocks_sip/day_aggs_v1/{yyyy}/{mm}/{yyyy-mm-dd}.csv.gz`, parallelized 16 workers). REST fallback when entitlement is missing: `GET /v2/aggs/grouped/locale/us/market/stocks/{date}?adjusted=true`, one call per trading day. Pattern matches `factor-research`. See [`../massive-flat-files/SKILL.md`](../massive-flat-files/SKILL.md) entitlement section. 4. **Pull splits and dividends** per [`references/corporate-action-adjustment.md`](./references/corporate-action-adjustment.md). `GET /v3/reference/splits?ticker={T}` and `GET /v3/reference/dividends?ticker={T}`. Massive's grouped aggs are split-adjusted by default (`adjusted=true`), so the skill does NOT re-apply the split ratio to OHLC; it does emit a cumulative adjustment factor as a separate column so the consumer can un-adjust if their backtester wants raw prints. 5. **Pull ticker reference** for sector enrichment. One call per ticker, parallelized. Adds `sic_code` and `sector` columns to the parquet. 6. **Detect edge cases.** IPO partial coverage (w