/consistency-coordination

name: consistency-coordination
description: This skill should be used when the user asks about the "CAP theorem", "PACELC", a "consistency model", "eventual vs strong consistency", "read-your-writes", "causal consistency", "quorum" or "R+W>N", "consensus", "Raft / Paxos", "leader election", "consistent hashing", a "distributed transaction", "2PC", or "saga". Use it whenever a design has multiple copies of data or coordinating nodes and a decision hinges on what a reader is guaranteed to see during replication lag, a network partition, or a node failure — even if the user doesn't say "consistency".

Consistency & Coordination

Decide what a reader is guaranteed to see when data lives on more than one node, and how independent nodes agree on a single answer. Get this wrong and the system either serves stale or conflicting data silently, or stalls the moment a network link drops — the most common and most punishing distributed-systems failure.

When to reach for this

Any time state is replicated, sharded, or coordinated across nodes: choosing a replication or quorum scheme, picking a consistency level, electing a leader, spreading keys across an elastic fleet (consistent hashing), or committing a change that spans services. Reach here the instant someone asks "what does a read see right after a write?" or "what happens during a partition?"

When NOT to

A single node with no replicas has nothing to coordinate — don't invoke quorums or consensus for it (YAGNI). Most apps tolerate seconds of staleness; reaching for strong consistency or distributed transactions when eventual consistency would do buys latency and operational pain for a guarantee no requirement asked for. The cheapest model that satisfies the invariant wins. Naming Raft or 2PC before a correctness requirement forces it is a red flag.

Clarify first

• **What breaks if a read is stale?** A wrong balance vs a slightly old like count

are different systems. (→ `requirements-scoping`.)

• **Must a user see their own writes immediately?** Read-your-writes is far cheaper

than global strong consistency.

• **What is the blast radius of a partition or lost region?** Drives the CAP/PACELC

choice. (GUIDE failure mode #1.)

• **Write contention and conflict shape** — single-writer per key, or concurrent

writers needing conflict resolution?

• **Latency budget** — synchronous coordination adds a round trip (or a

cross-region one); confirm the budget allows it. (→ `back-of-the-envelope`.)

The options

**Pick a consistency model** (the guarantee a read gets):

• **Strong / linearizable** — every read sees the latest committed write. Use when

an invariant must hold globally (balances, inventory, uniqueness).

• **Read-your-writes / monotonic** — a session sees its own writes and never goes

backward. Use for profile edits, "post then see your post".

• **Causal** — reads respect cause→effect order (a reply never precedes its

parent). Use for comments, chat, collaborative edits.

• **Eventual** — replicas converge "soon"; reads may lag. Use for like counts,

feeds, caches — anything where staleness is cheap.

**Pick how copies agree:**

• **Single-leader (primary)** — one node orders all writes; followers replicate.

Use when a clear write owner is acceptable; the common default.

• **Quorum (R + W > N)** — read/write to a majority; tune R and W per workload.

Use for leaderless stores needing tunable consistency and availability.

• **Consensus (Raft / Paxos)** — a majority agrees on a replicated log, even across

failures. Use for the control plane: leader election, config, metadata, locks.

**Coordinate a multi-key / multi-service change:**

• **Saga** — a sequence of local transactions with compensating undos. Use across

services where a global lock is impossible; accepts eventual consistency.

• **2PC / distributed transaction** — atomic all-or-nothing across nodes via a

coordinator. Use only when atomicity is mandatory and the cost is accepted.

**Spread keys across nodes — consistent hashing** (this skill owns it): map keys and nodes onto a hash ring; a key belongs to the next node clockwise. Adding or removing a node remaps only ~K/N keys instead of nearly all (as plain `hash(key) % N` does), avoiding a remap storm. Virtual nodes even out load and tame hotspots. It is the standard partitioning scheme for caches, leaderless stores, and LB backends. Mechanics in `references/deep-dive.md`.

Trade-offs

| Option | What it solves | What it worsens | Change it when | |---|---|---|---| | Strong/linearizable | Correctness; no stale reads | Latency (a round trip / quorum); unavailable under partition (CP) | Staleness becomes tolerable → relax to read-your-writes/eventual | | Read-your-writes | "See my own change" without global cost | Other users still see stale; needs session/sticky routing | A global invariant appears → go strong | | Eventual | Highest availability + lowest latency (AP) | Stale and conflicting reads; needs conflict resolution | An invariant can't tolerate divergence → stronger model | | Single-leader | Simple ordering; no write conflicts | Leader is a write SPOF; failover gap; reads from followers lag | Write throughput exceeds one node, or leader region lost → multi-leader/quorum | | Quorum (R+W>N) | Tunable consistency vs availability per call | Higher read+write cost; still needs conflict handling on concurrent writes | Even one quorum slow path is too costly → leader or local reads | | Consensus (Raft/Paxos) | Agreement that survives node loss | Majority required (loses availability below quorum); write latency; complex | The data plane needs it at scale → push coordination to a service | | Saga | Cross-service change without a global lock | No isolation; partial states visible; must design compensations | True atomicity is required → 2PC (and accept its cost) | | 2PC | Atomic multi-node commit | Coordinator is a SPOF; locks held across the vote; blocks on failure | Availability/throughput