/codspeed-setup-harness

name: codspeed-setup-harness
description: "Set up performance benchmarks and CodSpeed harness for a project. Use this skill whenever the user wants to create benchmarks, add performance tests, set up CodSpeed, configure codspeed.yml, integrate a benchmarking framework (criterion, divan, pytest-benchmark, vitest bench, go test -bench, google benchmark), or when the user says 'add benchmarks', 'set up perf tests', 'create a benchmark', 'benchmark this', or wants to measure performance of their code for the first time. Also trigger when the optimize skill needs benchmarks that don't exist yet."

Setup Harness

You are a performance engineer helping set up benchmarks and CodSpeed integration for a project. Your goal is to create useful, representative benchmarks and wire them up so CodSpeed can measure and track performance.

Step 1: Analyze the project

Before writing any benchmark code, understand what you're working with:

1. **Detect the language and build system**: Look at the project structure, package files (`Cargo.toml`, `package.json`, `pyproject.toml`, `go.mod`, `CMakeLists.txt`), and source files.

2. **Identify existing benchmarks**: Check for benchmark files, `codspeed.yml`, CI workflows mentioning CodSpeed or benchmarks.

3. **Identify hot paths**: Look at the codebase to understand what the performance-critical code is. Public API functions, data processing pipelines, I/O-heavy operations, and algorithmic code are good candidates.

4. **Check CodSpeed auth**: Ensure `codspeed auth login` has been run.

Step 2: Choose the right approach

Based on the language and what the user wants to benchmark, pick the right harness:

Language-specific harnesses (recommended when available)

These integrate deeply with CodSpeed and provide per-benchmark flamegraphs, fine-grained comparison, and simulation mode support.

| Language | Framework | How to set up | | ----------- | ------------------------------------------------ | -------------------------------------------------------------------------- | | **Rust** | divan (recommended), criterion, bencher | Add `codspeed-<framework>-compat` as dependency using `cargo add --rename` | | **Python** | pytest-benchmark | Install `pytest-codspeed`, use `@pytest.benchmark` or `benchmark` fixture | | **Node.js** | vitest (recommended), tinybench v5, benchmark.js | Install `@codspeed/<framework>-plugin`, configure in vitest/test config | | **Go** | go test -bench | No packages needed — CodSpeed instruments `go test -bench` directly | | **C/C++** | Google Benchmark | Build with CMake, CodSpeed instruments via valgrind-codspeed |

Exec harness (universal)

For any language or when you want to benchmark a whole program (not individual functions):

• Use `codspeed exec -m <mode> -- <command>` for one-off benchmarks
• Or create a `codspeed.yml` with benchmark definitions for repeatable setups

The exec harness requires no code changes — it instruments the binary externally. This is ideal for:

• Languages without a dedicated CodSpeed integration
• End-to-end benchmarks (full program execution)
• Quick setup when you just want to track a command's performance

Choosing simulation vs walltime mode

• **Simulation** (default for Rust, Python, Node.js, C/C++): Deterministic CPU simulation, <1% variance, automatic flamegraphs. Best for CPU-bound code. Does not measure system calls or I/O.
• **Walltime** (default for Go): Measures real execution time including I/O, threading, system calls. Best for I/O-heavy or multi-threaded code. Requires consistent hardware (use CodSpeed Macro Runners in CI).
• **Memory**: Tracks heap allocations. Best for reducing memory usage. Supported for Rust, C/C++ with libc/jemalloc/mimalloc.

Step 3: Set up the harness

Rust with divan (recommended)

1. Add the dependency:

cargo add divan
cargo add codspeed-divan-compat --rename divan --dev

2. Create a benchmark file in `benches/`:

// benches/my_bench.rs
use divan;

fn main() {
    divan::main();
}

#[divan::bench]
fn bench_my_function() {
    // Call the function you want to benchmark
    // Use divan::black_box() to prevent compiler optimization
    divan::black_box(my_crate::my_function());
}

3. Add to `Cargo.toml`:

[[bench]]
name = "my_bench"
harness = false

4. Build and run:

cargo codspeed build -m simulation --bench my_bench
codspeed run -m simulation -- cargo codspeed run --bench my_bench

Rust with criterion

1. Add dependencies:

cargo add criterion --dev
cargo add codspeed-criterion-compat --rename criterion --dev

2. Create benchmark in `benches/`:

use criterion::{criterion_group, criterion_main, Criterion};

fn bench_my_function(c: &mut Criterion) {
    c.bench_function("my_function", |b| {
        b.iter(|| my_crate::my_function())
    });
}

criterion_group!(benches, bench_my_function);
criterion_main!(benches);

3. Add to `Cargo.toml` and build/run same as divan.

Python with pytest-codspeed

1. Install:

pip install pytest-codspeed
# or
uv add --dev pytest-codspeed

2. Create benchmark tests:

# tests/test_benchmarks.py
import pytest

def test_my_function(benchmark):
    result = benchmark(my_module.my_function, arg1, arg2)
    # You can still assert on the result
    assert result is not None

# Or using the pedantic API for setup/teardown:
def test_with_setup(benchmark):
    data = prepare_data()
    benchmark.pedantic(my_module.process, args=(data,), rounds=100)

3. Run:

codspeed run -m simulation -- pytest --codspeed

Node.js with vitest (recommended)

1. Install:

npm install -D @codspeed/vitest-plugin
# or
pnpm add -D @codspeed/vitest-plugin

2. Co