/simulation-calibrator

Simulation Calibrator

Test and refine simulation accuracy with validation loops, bias detection, and continuous improvement frameworks.

Instructions

You are tasked with systematically calibrating simulations to ensure accuracy, reliability, and actionable insights. Follow this approach: **$ARGUMENTS**

1. Prerequisites Assessment

**Critical Calibration Context Validation:**

• **Simulation Type**: What kind of simulation are you calibrating?
• **Accuracy Requirements**: How precise does the simulation need to be?
• **Validation Data**: What real-world data can test simulation accuracy?
• **Decision Stakes**: How important are the decisions based on this simulation?
• **Update Frequency**: How often should calibration be performed?

**If context is unclear, guide systematically:**

Missing Simulation Context:
"What type of simulation needs calibration?
- Business Simulations: Market response, financial projections, strategic scenarios
- Technical Simulations: System performance, architecture behavior, scaling predictions
- Process Simulations: Operational workflows, resource allocation, timeline predictions
- Behavioral Simulations: Customer behavior, team dynamics, adoption patterns

Each simulation type requires different calibration approaches and validation methods."

Missing Accuracy Requirements:
"How accurate does your simulation need to be for effective decision-making?
- Mission Critical (95%+ accuracy): Safety, financial, or legal decisions
- Strategic Planning (80-95% accuracy): Investment, expansion, or major initiative decisions
- Operational Optimization (70-80% accuracy): Process improvement and resource allocation
- Exploratory Analysis (50-70% accuracy): Option generation and conceptual understanding"

2. Baseline Accuracy Assessment

**Establish current simulation performance levels:**

Historical Validation Framework

Simulation Accuracy Baseline:

Back-Testing Analysis:
- Compare simulation predictions to known historical outcomes
- Measure prediction accuracy across different time horizons
- Identify systematic biases and error patterns
- Assess prediction confidence calibration

Accuracy Metrics:
- Overall Prediction Accuracy: [percentage of correct predictions]
- Directional Accuracy: [percentage of correct trend predictions]
- Magnitude Accuracy: [percentage of predictions within acceptable error range]
- Timing Accuracy: [percentage of events predicted within correct timeframe]
- Confidence Calibration: [alignment between prediction confidence and actual accuracy]

Error Pattern Analysis:
- Systematic Biases: [consistent over/under-estimation patterns]
- Context Dependencies: [accuracy variations by scenario type or conditions]
- Time Horizon Effects: [accuracy changes over different prediction periods]
- Complexity Correlation: [accuracy relationship to scenario complexity]

Simulation Quality Scoring

Quality Assessment Framework:

Input Quality (25% weight):
- Data completeness and accuracy
- Assumption validation and documentation
- Expert input quality and consensus
- Historical precedent availability

Model Quality (25% weight):
- Algorithm sophistication and appropriateness
- Relationship modeling accuracy and completeness
- Constraint modeling and boundary definition
- Uncertainty quantification and propagation

Process Quality (25% weight):
- Systematic methodology application
- Bias detection and mitigation
- Stakeholder validation and feedback integration
- Documentation and reproducibility

Output Quality (25% weight):
- Prediction accuracy and reliability
- Insight actionability and clarity
- Decision support effectiveness
- Communication and presentation quality

Overall Simulation Quality Score = Sum of weighted component scores

3. Systematic Bias Detection

**Identify and correct simulation biases:**

Bias Identification Framework

Common Simulation Biases:

Cognitive Biases:
- Confirmation Bias: Seeking information that supports expected outcomes
- Anchoring Bias: Over-relying on first estimates or reference points
- Availability Bias: Overweighting easily recalled or recent examples
- Optimism Bias: Systematic overestimation of positive outcomes
- Planning Fallacy: Underestimating time and resource requirements

Data Biases:
- Selection Bias: Non-representative data samples
- Survivorship Bias: Only analyzing successful cases
- Recency Bias: Overweighting recent data points
- Historical Bias: Assuming past patterns will continue unchanged
- Measurement Bias: Systematic errors in data collection

Model Biases:
- Complexity Bias: Over-simplifying or over-complicating models
- Linear Bias: Assuming linear relationships where non-linear exist
- Static Bias: Not accounting for dynamic system changes
- Independence Bias: Ignoring correlation and interaction effects
- Boundary Bias: Incorrect system boundary definition

Bias Mitigation Strategies

Systematic Bias Correction:

Process-Based Mitigation:
- Multiple perspective integration and diverse expert consultation
- Red team analysis and devil's advocate approaches
- Assumption challenging and alternative hypothesis testing
- Structured decision-making and bias-aware processes

Data-Based Mitigation:
- Multiple data source integration and cross-validation
- Out-of-sample testing and validation dataset use
- Temporal validation across different time periods
- Segment validation across different contexts and conditions

Model-Based Mitigation:
- Ensemble modeling and multiple algorithm approaches
- Sensitivity analysis and robust parameter testing
- Cross-validation and bootstrap sampling
- Bayesian updating and continuous learning integration

4. Validation Loop Design

**Create systematic accuracy improvement processes:**

Multi-Level Validation Framework

Comprehensive Validation Approach:

Level 1: Internal Consistency Validation
- Logical consistency checking and constraint satisfaction
- Mathematical relatio