/3dgs-training-debugger

name: 3dgs-training-debugger
description: 'Diagnose and fix 3DGS training-time failures: NaN losses, OOM crashes, divergent optimization, floater
  artifacts, densification failures, hyperparameter sensitivity. Covers runtime debugging for vanilla 3DGS and 50+
  novel methods (deformable, MoE, physics-based, feed-forward). Detects 60 runtime failure patterns. Use when: 3DGS
  training crashes or produces poor results, loss is NaN/Inf, VRAM exhaustion, Gaussians explode or vanish, densification
  not working, convergence stalls, 训练调试/显存溢出/训练发散/浮点伪影.'
license: Apache-2.0
metadata:
  version: 1.0.0
  author: jaccen
  tags:
  - 3dgs
  - gaussian-splatting
  - training
  - debugging
  - oom
  - convergence
  - hyperparameters
  - distributed
  when_to_use:
  - 3DGS training crashes with OOM or CUDA error
  - Training loss becomes NaN or Inf
  - Gaussian count explodes or vanishes during training
  - Densification is not triggering or over-triggering
  - Training converges slowly or plateaus prematurely
  - Visual artifacts appear (floaters, blur, ghosting) after training
  - Checkpoint resume produces different results
  - Distributed training has inconsistent gradients or dead workers
  - Novel method (deformable/MoE/physics) diverges
  - 训练调试 / 显存溢出 / 训练发散 / 浮点伪影 / 密度控制异常
  platform-fields:
    user-invocable: true

3DGS Training Debugger

You are a senior 3DGS engineer who has trained hundreds of Gaussian Splatting models across vanilla 3DGS, deformable GS, feed-forward GS, SLAM-GS, and physics-based GS pipelines. Diagnose and fix training-time failures systematically.

Capabilities

• Diagnose training crashes (OOM, NaN/Inf loss, CUDA errors) with root-cause analysis
• Identify convergence failures (stalls, divergence, premature plateau)
• Debug densification failures (over/under-triggering, positional gradient issues)
• Diagnose visual artifacts from training logs (floaters, blur, ghosting, holes)
• Recommend hyperparameter adjustments with expected impact
• Guide distributed/multi-GPU training setup and debugging
• Troubleshoot checkpoint save/resume issues
• Address novel method stability (deformable GS, MoE-GS, physics-based GS, feed-forward GS)
• Detect 60 runtime failure patterns (see references/runtime-bug-patterns.md)

Relationship to Other Skills

This skill covers the **runtime training phase** — what happens AFTER code is written and BEFORE evaluation. It complements:

• **3dgs-code-reviewer**: Static code analysis (pre-training). Use code-reviewer first to catch implementation bugs, then use this skill to debug runtime issues.
• **3dgs-experiment-planner**: Experiment design (pre-training). Design experiments, then use this skill when training fails.
• **3dgs-engineering-guide**: Production deployment (post-training). This skill handles getting training TO completion.

Section 1: Training Monitoring Checklist

1.1 What to Monitor During Training

| Metric | Expected Behavior | Alert Threshold | Log Frequency | |--------|------------------|-----------------|---------------| | L1 loss | Decreasing, minor oscillation | Increase > 20% over 500 iters | Every 50 iters | | SSIM loss | Decreasing smoothly | Stagnant for 1000+ iters | Every 100 iters | | Total loss | Decreasing, plateau ~70-80% of training | NaN, Inf, or sudden spike | Every 50 iters | | PSNR (eval) | Increasing, plateau near end | Drop > 2dB between evals | Every 1000 iters | | Gaussian count | Growth phase (0-15k), then stable | Explosive growth (>10x) or vanishing | Every 500 iters | | VRAM usage | Stable with minor fluctuation during ADC | > 90% of total VRAM | Every 100 iters | | Gradient norms | Stable, < 1.0 typically | > 10.0 or exactly 0.0 | Every 100 iters | | Learning rate | Following schedule (warmup → cosine decay) | Unexpected reset or spike | Every 500 iters | | Active Gaussians | Growth then pruning equilibrium | All pruned (count → 0) | Every ADC cycle | | ADC trigger count | Periodic (every ~100 iters) | Never triggers or triggers every iter | Every ADC cycle |

1.2 Healthy Training Signature (Mip-NeRF 360, RTX 4090)

Reference trajectory for vanilla 3DGS on a typical Mip-NeRF 360 scene:

Iter     Loss    PSNR    Gaussians   VRAM     Notes
0        0.45    16.2    1(x SfM)    4.2 GB   Init from SfM points
500      0.22    20.1    12,000      5.1 GB   First ADC cycle
1000     0.15    23.5    45,000      7.8 GB   Rapid growth phase
2000     0.09    26.8    180,000     12.4 GB  Growth slowing
5000     0.05    29.2    350,000     14.2 GB  Near convergence
7000     0.04    30.1    380,000     14.5 GB  Fine-tuning
10000    0.03    30.4    390,000     14.6 GB  Final (shutting densif)
15000    0.03    30.5    395,000     14.6 GB  Densif frozen, opacity fine-tune
30000    0.025   30.6    395,000     14.6 GB  Final model

**Key signals**: Gaussian count should plateau around iter 10k-15k (when densification freezes), PSNR should still improve slightly afterward via opacity/SH refinement.

See `references/convergence-trajectories.md` for expected trajectories across datasets, scene types, and method variants.

Section 2: Failure Diagnosis Decision Tree

Start from the observed symptom and follow the branches.

SYMPTOM: Training crashed or NaN
│
├── NaN/Inf in loss?
│   ├── Check gradient norms → extremely large?
│   │   └── Possible: learning rate too high, gradient explosion
│   │       → Reduce lr to 1/10, add gradient clipping (max_norm=1.0)
│   ├── Check after ADC cycle → NaN appears right after densification?
│   │   └── Possible: NaN from clone/split, new Gaussian has bad scale/opacity
│   │       → Check scale clamping, opacity init values
│   ├── NaN from iteration 0?
│   │   └── Possible: bad initialization (zero covariance, SfM failure)
│   │       → Check point cloud, add covariance regularization
│   └── NaN in novel method (deformable/MoE)?
│       └── See Section 9: Novel Method Stability
│
├── CUDA OOM?
│   ├── During training (non-ADC)?
│   │   ├── Gaussi