/designing-distributed-system-tests

name: designing-distributed-system-tests
description: Use when designing a test plan for a distributed or stateful system — anything with persistence, replication, consensus, retries, idempotency, async messaging, multi-tenancy, or partial failure. Plans are claim-driven: investigate the product's claimed guarantees first, then design hypotheses and scenarios that try to falsify those claims under fault. Handles change-scoped plans (a commit / PR / feature) and project-wide plans (holistic, with existing-test inventory and gap analysis). Also use when asked to write a stability plan, fault matrix, release-validation plan, durability / partition / upgrade / crash-recovery / linearizability / deterministic-simulation plan, tenant isolation / authz / boundary plan, namespace isolation plan, fairness / noisy-neighbor plan, "what should we be testing", or "make a holistic test plan". Trigger even if the user just says "what should we test for this change", "are my tenants actually isolated", or "how do I test fairness across tenants / shards / queues".

Designing Distributed-System Tests

The default for testing distributed and stateful systems — write a few integration tests and call it done — finds a small fraction of the bugs that actually break these systems in production. This skill enforces an opinionated workflow: scope the change, generate failure-mode hypotheses that cover the categories the literature says matter most, pick techniques from a curated catalog, and emit a structured plan file that the executing-distributed-system-tests skill (or a human) can run.

Plan modes

This skill produces two shapes of plan. Decide which one applies before you start; the steps below branch on it.

• **Change-scoped** — the default. Use when the caller names a commit,

PR, branch-diff, or feature. The plan covers what *this change* could regress, scoped by its blast radius.

• **Project-wide** — use when the caller asks for a "release-validation

plan", "stability plan for the whole system", "test plan to enough coverage", "what should we be testing", or otherwise frames the request without a specific change. The plan covers what *the system* should be tested for, with an explicit inventory of existing tests and a gap analysis driving the new-scenario list.

If the framing is ambiguous, ask once before starting — the modes diverge enough that retrofitting one into the other wastes work.

Process

Follow these steps in order. Do not skip; the order matters because later steps depend on artifacts the earlier steps produce.

1. Scope the system

Read the project's entry points: `README`, `AGENTS.md` or `CLAUDE.md`, top-level `docs/`, any existing test-plan or runbook files. Note:

• Tenancy / isolation model
• Persistence model (what is durable, fsync contract)
• Replication / consensus protocol, quorum, leadership
• Ordering guarantee exposed to clients
• Network boundaries (which RPCs / streams)
• Retry / idempotency contract
• Observability (logs, metrics, traces) available to an oracle

Write this as a one-paragraph SUT model. If anything is ambiguous from the repo, ask the user before proceeding — do not invent guarantees.

1b. Extract claims and guarantees

A good test plan exists to falsify what the product *claims*. Before generating hypotheses, write down what the SUT promises its users. This is the spine the rest of the plan hangs off — every hypothesis, every scenario, every oracle should be traceable back to a claim it either confirms or refutes.

Sources to mine:

• README "guarantees" / "what we offer" sections
• API docs / reference manuals
• ARCHITECTURE / DESIGN docs (claims about consistency, durability,

replication, fault tolerance)

• Public blog posts, talks, marketing material (if any)
• The code itself: function names, doc-comments on public APIs,

error types (`IdempotencyConflict`, `StaleRead`, etc.) imply guarantees the system claims to enforce

• Existing test names (a test called `linearizable_under_partition`

implies a linearizability claim under partition)

Categorise each claim:

• **Safety** — "the system never returns a stale read", "no

acknowledged write is ever lost", "linearizable per key"

• **Liveness** — "every accepted operation eventually commits",

"leader election completes within N seconds of crash"

• **Durability** — "fsync'd writes survive crash", "replicated

writes survive single-AZ loss"

• **Performance / SLO** — "p99 append latency ≤ X ms at Y ops/s

per session"

• **Operational** — "rolling upgrade is non-disruptive",

"configuration changes are atomic"

• **Idempotency / dedup** — "same idempotency key never produces

two committed effects"

• **Isolation** — "tenant A's reads never observe tenant B's writes",

"no read returns data from a transaction that has not yet committed"

• **Ordering** — "consumers always see messages in the order the

producer sent them", "every reader sees a prefix of the global log order"

• **Membership** — "a node that fails its liveness probe is removed

from the cluster membership view within N seconds", "every joined member appears in the membership table exactly once"

• **Boundary** — access-boundary semantics: "tenant A's data is never

reachable from tenant B on any surface", "a request scoped to namespace X never routes to namespace Y". Subsumes tenancy / authz / namespace / routing / multi-protocol; do not file those as separate categories. Triggers the §7.M.S surface-decomposition discipline (see step 3 and `references/boundary-and-isolation-testing.md`).

• **Fairness** — per-group performance and noisy-neighbor isolation:

"no tenant can starve another for throughput", "one shard's load does not blow another shard's p99". Group can be tenant, shard, queue, partition, region, priority class, user, table, or workload class. Also triggers §7.M.S.

If the project does NOT explicitly document a claim that appears in the code, write it as an *inferred