/establish-empirical-baseline

name: establish-empirical-baseline
description: "Establish a fair empirical baseline by inventorying methods, extracting comparable performance, normalizing conditions/compute, checking discrepancies, and estimating progress/headroom."

establish-empirical-baseline

Purpose

Establish a fair empirical baseline by inventorying methods, extracting comparable performance, normalizing conditions/compute, checking discrepancies, and estimating progress/headroom.

Input contract

required: [method_records, benchmark_or_task, performance_measure]
optional: [historical_series, compute_metadata, condition_schema]
constraints: [comparability fields and source provenance required]

Execution protocol

Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.

1. You MUST load skill `inventory-reference-items` to inventory methods and comparison records. 2. You MUST load skill `extract-evidence-record` to extract performance evidence. You MUST load skill `audit-reporting-quality` to audit missing or ambiguous reporting. You MUST load skill `normalize-comparison-scale` to normalize units, data, compute, and evaluation protocol. 3. You MUST load skill `detect-performance-discrepancy` to detect comparison discrepancies. You MUST load skill `estimate-performance-headroom` to estimate headroom. You MUST load skill `analyze-temporal-trajectory` to analyze progress and leaderboard dynamics. You MUST load skill `check-dominance` to identify dominated and incomparable records. 4. Synthesize the baseline with uncertainty and known incomparable records. If the resulting methods or baselines require explicit comparative selection, consider `rank-candidates` as the next tactic.

Output contract

produces: [method_inventory, normalized_baseline, discrepancy_report, progress_curve, headroom_estimate]
delta_fields: [findings, evidence_updates, uncertainties, decisions, open_questions]

Thresholds and quality gates

• Baseline acquisition gates are relative to a declared eligible universe: record numerator, denominator, batch increment, stopping reason, and source references for each ratio.
• `method-inventory`: method coverage ratio reaches a justified floor over the eligible method universe.
• `performance-extraction`: comparable-record coverage ratio reaches a justified floor over eligible records.
• `condition-standardization`: complete condition-vector ratio reaches a justified floor over comparable records.
• `discrepancy-analysis`: score-pair coverage ratio reaches a justified floor over eligible comparison pairs.
• `progress-quantification`: historical-time coverage and independent-source ratio reach justified floors; stop when added periods no longer change the trajectory conclusion.
• Across modes, report marginal information gain and saturation state when added records or periods no longer change the baseline conclusion.
• Normalization must expose condition, compute, metric, and unit transformations.

Failure and counterexamples

Do not call a baseline fair when conditions are missing, metrics are incomparable, or leaderboard values are copied without protocol verification. Mark headroom unknown when the historical series is below its floor.

Provenance map

9 architecture `old` entries: baseline, inventory, extraction, standardization, discrepancy, progress, leaderboard, normalization, curve construction. Repeated reporting prose compressed.

Legacy context checkpoint / Delta notes

Append method IDs, normalized records, excluded records with reasons, discrepancy pairs, progress model, and headroom uncertainty.

Preserved source criteria ledger

| source | source line | kind | source criterion | |---|---:|---|---| | baseline-establishment | 28 | textual | \\| Check reproducibility / discrepancies \\| discrepancy-analysis \\| | | baseline-establishment | 40 | textual | \\| discrepancy-analysis \\| Identify discrepancies between reported and reproducible scores \\| | | baseline-establishment | 58 | textual | \\| reproducibility-checklist-audit \\| Assess paper against ML Reproducibility Checklist \\| | | baseline-establishment | 70 | numeric-table | \\| method-inventory \\| 50 \\| 0 \\| 60 \\| | | baseline-establishment | 71 | numeric-table | \\| performance-extraction \\| 30 \\| 150 \\| 40 \\| | | baseline-establishment | 72 | numeric-table | \\| condition-standardization \\| 20 \\| 60 \\| 30 \\| | | baseline-establishment | 73 | numeric-table | \\| discrepancy-analysis \\| 15 \\| 45 \\| 30 \\| | | baseline-establishment | 74 | numeric-table | \\| progress-quantification \\| 30 \\| 100 \\| 40 \\| | | baseline-establishment | 75 | numeric-table | \\| **TOTAL** \\| **145** \\| **355** \\| **200** \\| | | method-inventory | 23 | numeric-table | \\| Methods discovered \\| 30 \\| 50 \\| | | method-inventory | 24 | numeric-table | \\| Web searches \\| 40 \\| 60 \\| | | method-inventory | 25 | numeric-table | \\| Papers consulted \\| 20 \\| 40 \\| | | method-inventory | 30 | textual | <HARD-GATE> | | method-inventory | 33 | numeric-table | \\| Methods discovered \\| 0 \\| 50 \\| BLOCKED \\| | | method-inventory | 34 | numeric-table | \\| Web searches used \\| 0 \\| 60 \\| - \\| | | method-inventory | 35 | numeric-table | \\| Papers consulted \\| 0 \\| 40 \\| - \\| | | method-inventory | 36 | numeric-table | \\| Leaderboard sources \\| 0 \\| 5 \\| - \\| | | method-inventory | 37 | numeric-table | \\| Citation chains traced \\| 0 \\| 10 \\| - \\| | | method-inventory | 38 | textual | </HARD-GATE> | | method-inventory | 41 | numeric | Cannot exit until methods_discovered >= 40 (80% of target). | | performance-extraction | 18 | textual | Extract structured performance data from papers, leaderboards, and reproducibility studies. Each data point is a (Task, Dataset, Metric, Score, Conditions) tuple with full provenance. Prioritizes primary sources (original papers) but cross-references against leaderboards and third-p