/checkpoint-and-recover

name: checkpoint-and-recover
description: Checkpoint state before risky operations, detect anomalies, and recover
  gracefully
version: 1.0.0
category: experiment-execution
type: tactic
orchestrates:
- execution-monitoring
- result-collection
dependencies:
  sops:
  - execution-monitoring
  - result-collection

Tactic: Checkpoint and Recover

Orchestration Pattern

FUNCTION checkpoint_and_recover(task, execute_fn):
    // Pre-execution checkpoint
    checkpoint = {
        timestamp: now(),
        task_id: task.id,
        state: capture_current_state(),
        files_modified: [],
        outputs_produced: []
    }
    save_checkpoint(checkpoint)
    
    TRY:
        // Execute with monitoring
        monitor = SPAWN execution-monitoring(task)
        result = execute_fn(task)
        
        // Post-execution validation
        IF monitor.anomalies_detected:
            RAISE AnomalyError(monitor.anomalies)
        END
        
        // Validate result integrity
        validated = SPAWN result-collection(result, task.success_criterion)
        
        IF validated.complete AND validated.consistent:
            // Success — archive checkpoint (keep for audit trail)
            archive_checkpoint(checkpoint)
            RETURN {status: DONE, result: validated}
        ELSE:
            // Partial success — decide whether to keep or rollback
            IF validated.partial_value > threshold:
                archive_checkpoint(checkpoint)
                RETURN {status: PARTIAL, result: validated, missing: validated.gaps}
            ELSE:
                restore_state(checkpoint)
                RETURN {status: ROLLED_BACK, reason: validated.failure_reason}
            END
        END
        
    CATCH error:
        // Failure — diagnose and recover
        diagnosis = diagnose_failure(error, checkpoint, task)
        
        SWITCH diagnosis.severity:
            CASE TRANSIENT:
                // Retry without rollback (e.g., network timeout)
                RETURN {status: RETRY, reason: diagnosis}
                
            CASE CORRUPTING:
                // Rollback to checkpoint
                restore_state(checkpoint)
                RETURN {status: ROLLED_BACK, reason: diagnosis}
                
            CASE FATAL:
                // Rollback and escalate
                restore_state(checkpoint)
                RETURN {status: FATAL, reason: diagnosis, escalate: true}
        END
    END
END

Decision Criteria

| Condition | Action | |-----------|--------| | Task modifies existing files | MUST checkpoint before | | Task is read-only/analysis | Checkpoint optional | | Anomaly detected during execution | Pause, diagnose, decide | | Result partially valid | Keep if value > threshold | | Result invalid | Rollback to checkpoint | | Transient error (timeout, rate limit) | Retry without rollback | | Corrupting error (bad state) | Rollback then retry | | Fatal error (impossible task) | Rollback and escalate |

Checkpoint Contents

A checkpoint captures:

• Timestamp and task ID
• File system state (modified files' contents before modification)
• Execution context (variables, intermediate results)
• Dependencies state (which tasks were complete)

Recovery Strategies

1. **Retry**: Same task, same parameters (for transient failures) 2. **Retry with modification**: Same task, adjusted parameters (for NEEDS_CONTEXT) 3. **Rollback and skip**: Restore state, mark task BLOCKED, continue 4. **Rollback and escalate**: Restore state, report to orchestrator for human decision

<!-- BEGIN available-tables (generated) -->

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | execution-monitoring | Monitor execution progress, detect anomalies, and report status | | result-collection | Collect experiment outputs — metrics, logs, artifacts — into structured result set |

<!-- END available-tables (generated) -->