/cpu-gpu-performance

name: cpu-gpu-performance
description: Establishes CPU/GPU baselines before resource-intensive operations. Use before builds, training runs, or any task that pins cores or GPUs for over a minute.
alwaysApply: false
progressive_loading: true
dependencies:
  hub:
  - token-conservation
  modules: []

CPU/GPU Performance Discipline

When To Use

• At the beginning of every session (auto-load alongside `token-conservation`).
• Whenever you plan to build, train, or test anything that could pin CPU cores

or GPUs for more than a minute.

• Before retrying a failing command that previously consumed significant resources.

When NOT To Use

• Simple operations with no resource impact
• Quick single-file operations

Required Progress Items

1. `cpu-gpu-performance:baseline` 2. `cpu-gpu-performance:scope` 3. `cpu-gpu-performance:instrument` 4. `cpu-gpu-performance:throttle` 5. `cpu-gpu-performance:log`

Step 1: Establish Current Baseline

• Capture current utilization:
• `uptime`
• `ps -eo pcpu,cmd | head`
• `nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv`

Note which hosts/GPUs are already busy.

• Record any CI/cluster budgets (time quotas, GPU hours) before launching work.
• Set a per-task CPU minute / GPU minute budget that respects those limits.

Step 2: Narrow the Scope

• Avoid running "whole world" jobs after a small fix. Prefer diff-based

or tag-based selective testing:

• `pytest -k`
• Bazel target patterns
• `cargo test <module>`
• Batch low-level fixes so you can validate multiple changes with a single targeted command.
• For GPU jobs, favor unit-scale smoke inputs or lower epoch counts before

scheduling the full training/eval sweep.

Step 3: Instrument Before You Optimize

• Pick the right profiler/monitor:
• CPU work:
• `perf`
• `intel vtune`
• `cargo flamegraph`
• language-specific profilers
• GPU work:
• `nvidia-smi dmon`
• `nsys`
• `nvprof`
• DLProf
• framework timeline tracers
• Capture kernel/ops timelines, memory footprints, and data pipeline latency

so you have evidence when throttling or parallelizing.

• Record hot paths and I/O bottlenecks in notes so future reruns can jump straight to the culprit.

Step 4: Throttle and Sequence Work

• Use `nice`, `ionice`, or Kubernetes/Slurm quotas to prevent starvation of shared nodes.
• Chain heavy tasks with guardrails:
• Rerun only the failed test/module
• Then (optionally) escalate to the next-wider shard
• Reserve the full suite for the final gate
• Stagger GPU kernels (smaller batch sizes or gradient accumulation) when memory

pressure risks eviction; prefer checkpoint/restore over restarts.

Step 5: Log Decisions and Next Steps

Conclude by documenting the commands that were run and their resource cost (duration, CPU%, GPU%), confirming whether they remained within the per-task budget. If a full suite or long training run was necessary, justify why selective or staged approaches were not feasible. Capture any follow-up tasks, such as adding a new test marker or profiling documentation, to simplify future sessions.

Output Expectations

• Brief summary covering:
• baseline metrics
• scope chosen
• instrumentation captured
• throttling tactics
• follow-up items
• Concrete example(s) of what ran (e.g.):
• "reran `pytest tests/test_orders.py -k test_refund` instead of `pytest -m slow`"
• "profiled `nvidia-smi dmon` output to prove GPU idle time before scaling"

Exit Criteria

• [ ] `uptime` and `ps` baseline captured and recorded before any

build, training run, or test suite starts

• [ ] Scope narrowed to diff-based or tag-based targets (e.g.,

`pytest -k`, `cargo test <module>`); full-suite justification documented if selective approach was not feasible

• [ ] Output summary includes: duration, CPU% or GPU% consumed, and

whether the run stayed within the per-task budget

• [ ] Any follow-up tasks (new test markers, profiling docs) written

to a todo or issue so they survive the session