/designing-distributed-system-tests

name: designing-distributed-system-tests
description: >-
  Design claim-driven test plans for distributed or stateful systems with persistence, replication, consensus, retries, idempotency, async messaging, multi-tenancy, or partial failure. Supports change-scoped and project-wide plans, including existing-test inventories and gap analysis. Use for stability, release-validation, durability, partition, crash-recovery, upgrade, linearizability, deterministic-simulation, SQL semantics or compatibility, tenant isolation, authz, namespace boundaries, fairness, and noisy-neighbor testing. Investigate claimed guarantees and design scenarios to falsify them during normal operation and under relevant faults. Also use for "what should we test for this change", "are my tenants isolated", or "make a holistic test plan".

Designing Distributed-System Tests

The default for testing distributed and stateful systems — write a few integration tests and call it done — finds a small fraction of the bugs that actually break these systems in production. This skill enforces an opinionated workflow: scope the change, generate failure-mode hypotheses that cover the categories the literature says matter most, pick techniques from a curated catalog, and emit a structured plan file that the executing-distributed-system-tests skill (or a human) can run.

Plan modes

This skill produces two shapes of plan. Decide which one applies before you start; the steps below branch on it.

• **Change-scoped** — the default. Use when the caller names a commit,

PR, branch-diff, or feature. The plan covers what *this change* could regress, scoped by its blast radius.

• **Project-wide** — use when the caller asks for a "release-validation

plan", "stability plan for the whole system", "test plan to enough coverage", "what should we be testing", or otherwise frames the request without a specific change. The plan covers what *the system* should be tested for, with an explicit inventory of existing tests and a gap analysis driving the new-scenario list.

If the framing is ambiguous, ask once before starting — the modes diverge enough that retrofitting one into the other wastes work.

Process

Follow these steps in order. Do not skip; the order matters because later steps depend on artifacts the earlier steps produce.

1. Scope the system

Read the project's entry points: `README`, `AGENTS.md` or `CLAUDE.md`, top-level `docs/`, any existing test-plan or runbook files. Note:

• Tenancy / isolation model
• Persistence model (what is durable, fsync contract)
• Replication / consensus protocol, quorum, leadership
• Ordering guarantee exposed to clients
• Network boundaries (which RPCs / streams)
• Retry / idempotency contract
• Observability (logs, metrics, traces) available to an oracle

Write this as a one-paragraph SUT model. If anything is ambiguous from the repo, ask the user before proceeding — do not invent guarantees.

1b. Extract claims and guarantees

A good test plan exists to falsify what the product *claims*. Before generating hypotheses, write down what the SUT promises its users. This is the spine the rest of the plan hangs off — every hypothesis, every scenario, every oracle should be traceable back to a claim it either confirms or refutes.

Sources to mine:

• README "guarantees" / "what we offer" sections
• API docs / reference manuals
• ARCHITECTURE / DESIGN docs (claims about consistency, durability,

replication, fault tolerance)

• Public blog posts, talks, marketing material (if any)
• The code itself: function names, doc-comments on public APIs,

error types (`IdempotencyConflict`, `StaleRead`, etc.) imply guarantees the system claims to enforce

• Existing test names (a test called `linearizable_under_partition`

implies a linearizability claim under partition)

Categorise each claim:

• **Safety** — "the system never returns a stale read", "no

acknowledged write is ever lost", "linearizable per key"

• **Liveness** — "every accepted operation eventually commits",

"leader election completes within N seconds of crash"

• **Durability** — "fsync'd writes survive crash", "replicated

writes survive single-AZ loss"

• **Performance / SLO** — "p99 append latency ≤ X ms at Y ops/s

per session"

• **Operational** — "rolling upgrade is non-disruptive",

"configuration changes are atomic"

• **Idempotency / dedup** — "same idempotency key never produces

two committed effects"

• **Isolation** — "tenant A's reads never observe tenant B's writes",

"no read returns data from a transaction that has not yet committed"

• **Ordering** — "consumers always see messages in the order the

producer sent them", "every reader sees a prefix of the global log order"

• **Membership** — "a node that fails its liveness probe is removed

from the cluster membership view within N seconds", "every joined member appears in the membership table exactly once"

• **Boundary** — access-boundary semantics: "tenant A's data is never

reachable from tenant B on any surface", "a request scoped to namespace X never routes to namespace Y". Subsumes tenancy / authz / namespace / routing / multi-protocol; do not file those as separate categories. Triggers the §7.M.S surface-decomposition discipline (see step 3 and `references/boundary-and-isolation-testing.md`).

• **Fairness** — per-group performance and noisy-neighbor isolation:

"no tenant can starve another for throughput", "one shard's load does not blow another shard's p99". Group can be tenant, shard, queue, partition, region, priority class, user, table, or workload class. Also triggers §7.M.S.

If the project does NOT explicitly document a claim that appears in the code, write it as an *inferred* claim and mark it as such — inferred claims are still testable, and surfacing them often catches places where the docs lie or are silent about real guarantees the implementation depends on.

Bind safety-critical claims to a version and config