/memory-benchmark

name: memory-benchmark
description: How to benchmark and analyze memory usage in Turso using the memory-benchmark crate and dhat heap profiler. Use this skill whenever the user mentions memory usage, memory profiling, allocation tracking, heap analysis, memory regression, memory benchmarking, dhat, or wants to understand where memory is being allocated during SQL workloads. Also use when investigating memory growth in WAL or MVCC mode. IMPORTANT - If you modify the perf/memory crate (add profiles, change CLI flags, change output format, etc.), update this skill document to reflect those changes so it stays accurate for future agents.

Memory Benchmarking & Analysis

The `perf/memory` crate benchmarks memory usage of SQL workloads under WAL and MVCC journal modes. It uses `dhat` as the global allocator to track every heap allocation, and `memory-stats` for process-level RSS snapshots.

It also contains a `stack-report` helper binary for stack-usage investigations. That binary runs a SQL payload with the `stacker` feature enabled and captures `turso_stack` tracing events in-process, aggregating structured tracing fields instead of parsing stderr log text.

Location

• Benchmark crate: `perf/memory/`
• CodSpeed bench crate: `perf/memory/codspeed/` (CI allocation regression tracking)
• Analysis script: `perf/memory/analyze-dhat.py`
• dhat output: `dhat-heap.json` (written to CWD after each run)

The crate is split into a library and binaries. The workload engine lives in `memory_benchmark::workload` (`run_workload`, `WorkloadConfig`, `WorkloadObserver`, the `JournalMode`/`WorkloadProfile` enums and `create_profile`); the `memory-benchmark` bin is a thin CLI over it that adds dhat/RSS measurement. Randomized profiles (`read-heavy`, `mixed`) use a fixed RNG seed (`profile::WORKLOAD_RNG_SEED`) so workloads are identical across runs.

MVCC Memory Per Row

The `mvcc-row-memory` binary measures the heap that one MVCC transaction keeps for each row it writes: one `INSERT`, `UPDATE` or `DELETE` over every row of `t(id INTEGER PRIMARY KEY, v BLOB)`. `update` and `delete` change rows that a checkpoint already wrote to the B-tree. Use it to measure changes to MvStore data structures (version chains, write sets, tombstones).

cargo run -p memory-benchmark --bin mvcc-row-memory -- \
  --statement insert|update|delete --rows 100000 --payload-bytes 100 \
  --mode mvcc|wal --dhat-file /tmp/rows.json
python3 perf/memory/analyze-dhat.py /tmp/rows.json --sort-by eb --top 10

The JSON report has `uncommitted_bytes_per_row` (before `COMMIT`), `statement_peak_bytes_per_row`, `commit_peak_bytes_per_row`, `committed_bytes_per_row` (after `COMMIT`, with automatic checkpoints and GC off) and `statement_allocations_per_row`. The byte values are requested bytes, so a debug build gives the same numbers as an optimized build. The dhat file's "At exit" values are the live allocations at the committed measurement. See `perf/memory/README.md` for details.

FTS Query Allocations

The `fts-memory` binary and `perf/memory/codspeed/benches/fts_queries.rs` share `memory_benchmark::fts`, independently of the existing SQL profile runner. See `perf/memory/README.md` for commands, corpus details, and measurement limits.

cargo run -p memory-benchmark --features fts --bin fts-memory -- \
  --query common --state warm --documents 10000 --queries 100 \
  --dhat-file /tmp/fts-warm.json > /tmp/fts-warm-report.json
python3 perf/memory/analyze-dhat.py /tmp/fts-warm.json --modules --top 20
cargo test -p memory-benchmark-codspeed --features fts --bench fts_queries -- --test

Use debug runs for correctness only; use `--profile bench-profile` for optimized dhat investigations with debug information and no LTO. Query cases are `rare`, `common`, `and`, `or`, `phrase`, and `ranked`. State `first` requires one query (the default); `warm` defaults to 100 queries on one persistent connection after one unmeasured warm-up. The default corpus has 10000 documents; 1000 and 100000 are useful local comparison sizes. WAL and one connection remain the defaults.

For full transaction lifecycles, pass `--mode mvcc --connections 2 --transactions-per-connection 10 --queries-per-transaction 2`. This measures 20 transactions and 40 queries, including `BEGIN CONCURRENT` and `COMMIT`. WAL uses `BEGIN`. Explicit transaction counts conflict with `--queries`; query counts are otherwise per connection. All transactions begin before query workers start, and all workers finish before commits. Connections persist across rounds. One connection covers repeated transactions; two/four cover overlapping snapshots. Warm-up runs one complete transaction with one query per connection when explicit transactions are selected. Reports include completed transaction/query counts and `max_active_transactions`; heap totals aggregate all connections and scheduling.

`FtsWorkload` emits `setup`, `open`, `warmup`, `run`, `cleanup`, and `done` through `FtsObserver`. The CLI's `DhatObserver` starts profiling on `run` and stops on `cleanup`, before connections are dropped. Each transition emits a JSON phase event to stderr, also saved in the final report's `phases` array. stderr includes dhat diagnostics; filter phase events with `jq -Rc 'fromjson? | select(.event == "phase")'`. Elapsed event times include profiler overhead; check exit status for success. `after_batch` samples live heap after a query per connection or after all commits in one transaction round. Worker errors are joined before rollback and cleanup.

Setup, database opening, warm-up, and cleanup are excluded. Heap totals, peaks, retained bytes, and per-query live-byte samples track query-phase allocations, not caches allocated before profiling. RSS after profiling includes dhat's own stack/report overhead. JSON goes to stdout and allocation stacks to `--dhat-file`.

The Divan target uses the workspace's CodSpeed-compatible Divan dependency, without dhat's allocator, and runs in the `ft