/run-napkin-math-pipeline

name: run-napkin-math-pipeline
description: Use when the user wants to run the napkin-math pipeline end-to-end on a PlanExe report, or resume a partially populated output directory by filling in only the missing stages. Orchestrates digest preparation, parameter extraction, validation, bounds, calculations, scenarios, Monte Carlo, and assessment rendering. Never copies artifacts forward from prior runs, and never re-runs a stage whose output is already on disk.

Run the Napkin-Math Pipeline

Overview

End-to-end orchestrator for the pipeline documented in `experiments/napkin_math/README.md`. Decides which stages to run based on what is already present in the target output directory, then walks forward through the remaining stages one at a time.

PlanExe source reports live under `/Users/neoneye/git/PlanExe-web/` (e.g. `/Users/neoneye/git/PlanExe-web/20251114_paperclip_automation/`). Outputs land under `experiments/napkin_math/output/<version>/<plan-slug>/`.

What the orchestrator does and does not do

The orchestrator is **dispatch-only**. It looks at filenames in the target directory, decides which stage is missing, and delegates the work to the appropriate sibling skill (via a subagent) or Python runner. It does **not** read the content of large input files into its own context.

**Do not read into orchestrator context:**

• `extract_parameters_input.md` (the digest — typically ~25K tokens)
• `compress_*.md` or `compress_*_raw.json` files
• The raw PlanExe HTML report
• The full `parameters.json`, `bounds.json`, `scenarios.json`, `montecarlo.json` (except quick `jq`/Python extraction of a single field)

**OK to read into orchestrator context:**

• Directory listings (filenames only)
• `validation.json` exit status (via Python one-liner, not full file)
• `montecarlo.json` per-threshold `probability` values (via Python one-liner)
• Sibling skill SKILL.md / system-prompt.txt **only when updating the skill itself**

Pulling a digest into orchestrator context defeats the architecture. Each stage's sibling skill knows how to read its own inputs; the orchestrator's job is to invoke that skill, not to do its work.

Per-stage delegation

For each missing stage, dispatch as follows. Pass the agent the file paths and a one-line task; do not paste file contents into the prompt.

| Stage | How to dispatch | |---|---| | 0. Digest | `Bash` → `prepare_extract_input.py --planexe-dir <PlanExe-web/...> --output-dir <target>` | | 1. Parameters | `Agent` with the sibling skill name `extract-parameters-from-digest`; prompt: "Read `<target>/extract_parameters_input.md` per `system-prompt.txt`, write the result to `<target>/parameters.json`." | | 2. Validation | `Bash` → `validate_parameters.py --parameters … --output …` | | 3. Bounds | `Agent` with `generate-bounds`; prompt: "Read `<target>/parameters.json` per the generate-bounds rules, write `<target>/bounds.json`." | | 4. Calculations | `Agent` with `generate-calculations`; prompt: "Read `<target>/parameters.json`, write `<target>/calculations.py` per the skill rules." | | 5. Scenarios | `Agent` with `run-scenarios`; prompt: "Read parameters.json, bounds.json, calculations.py, write `<target>/scenarios.json`." | | 6. MC settings | Small hand-written JSON file based on `parameters.json` — extract `output_name`s via Python one-liner, write thresholds with `>= 0`. No content reading needed. | | 7. Monte Carlo | `Bash` → `run_monte_carlo.py …` | | 8. Assessment | `Bash` → `summarize_assessment.py …` |

After each agent returns, verify the output file exists and is non-empty, then move on. If validation fails after Stage 1, dispatch a repair agent with the validation report — do not pull the parameters into context yourself.

Why resume mode exists

Resume is not a convenience. It is the experimental control for this pipeline.

Several stages — digest preparation, parameter extraction, bounds, calculations, scenarios — are LLM-driven and therefore non-deterministic. If every run regenerated them from scratch, every downstream change would be confounded with upstream sampling noise, and the user could not tell whether a different `assessment.md` came from a skill/prompt tweak or just from a different roll of the LLM.

The intended workflow is:

1. On the first run for a plan, populate the directory once. The `extract_parameters_input.md` digest (Stage 0) becomes the pinned starting point. 2. On subsequent runs — typically after editing a skill prompt, a validator rule, or a runner — the user deletes only the artifacts downstream of the change, and resumes from the highest-numbered stage whose output is still present. 3. Because the input to the first re-run stage is byte-identical to the prior run, any downstream difference can be attributed to the tweak under test.

The corollary is strict: **never re-run a stage whose output is already on disk.** Doing so silently changes the input that every later stage sees, and destroys the comparability the user is trying to preserve. If the user wants a stage re-run, they will delete its output file first; absence of the file is the signal to run, presence is the signal to leave alone.

The two cardinal rules

**1. Never copy a pipeline artifact from a different output directory into the target directory.** Skill prompts, validator rules, and runner logic evolve between versions; copying `output/<vN-1>/<plan>/<file>` into `output/<vN>/<plan>/` silently embeds the old rules into the new run. If at any point you are about to type `cp /…/v{N-1}/<plan>/<file> $D/`, stop and run the stage that produces `<file>` instead.

**2. Never re-run a stage whose output is already on disk.** The user's A/B comparability depends on the input to each stage being byte-identical to the prior run, up to the first deleted artifact. Re-running a present stage (especially Stage 0, the LLM-driven digest) re-rolls the dice on its input distribution and confounds every downstream comparison. The only f