/data-storage

name: data-storage
description: This skill should be used when the user asks "SQL or NoSQL", "which database", how to design a "data model" or "schema design", picks an "indexing" strategy, needs "sharding" or "partitioning", sets up "replication" (leader-follower / multi-leader), defines a "primary key"/"sort key", asks whether to "denormalize", or weighs "polyglot persistence". Use it whenever a design must decide where records live, how they are keyed and accessed, and how the store scales past one node — even if the user just says "store the data".

Data Storage

Choose where records live, how they are keyed and queried, and how the store grows past a single machine. Storage is the hardest layer to change later: a wrong data model or shard key calcifies into a scaling ceiling, and getting replication wrong silently serves stale or lost data.

When to reach for this

Any system that persists state: picking SQL vs NoSQL, designing a schema and its access paths, adding indexes, splitting a hot table, distributing data across nodes (sharding/partitioning), adding read replicas, or deciding what to denormalize. Reach here the moment "store the data" needs a concrete key and query shape.

When NOT to

Don't shard, add replicas, or reach for NoSQL before a number forces it (YAGNI). A single well-indexed relational node handles ~1k QPS and tens of GB to low TB comfortably — most systems never outgrow it. Sharding multiplies operational cost and breaks joins/transactions; add it only when one node's write throughput or dataset size is genuinely exceeded (→ `back-of-the-envelope`). Caching reads (→ `caching`) and adding read replicas are cheaper first moves than sharding.

Clarify first

Answer these before choosing a store or topology — they decide the design:

• **Data shape & relationships** — flat key-value? rich relations needing joins?

document blobs? a graph of connections? Drives SQL vs NoSQL.

• **Access patterns** — *how* is data read and written, not just where it lives.

Point lookups by key, range scans, ad-hoc queries, aggregations? Model the store around the queries it must serve.

• **Read:write ratio & scale** — QPS each way, total size now and at retention.

(→ `back-of-the-envelope` for QPS, storage, and shard counts.)

• **Consistency need** — must reads see the latest write, or is eventual OK? Are

multi-record transactions required? (CAP/consistency theory → `consistency-coordination`.)

• **Latency & durability targets** — p99 read/write budget, and how much recent

data the system can afford to lose on a node failure.

The options

**Relational (SQL — Postgres, MySQL):** strict schema, joins, ACID transactions. *Use when* data is relational, integrity matters, and queries are varied/ad-hoc — the safe default until a number rules it out.

**Document (MongoDB, etc.):** flexible schema, self-contained JSON-ish documents queried by structure. *Use when* records are read/written as a whole and the schema evolves; relationships are few.

**Key-value (Redis, DynamoDB, Riak):** O(1) get/put by key, no rich queries. *Use when* access is purely by a known key and massive throughput is needed.

**Wide-column (Cassandra, Bigtable, HBase):** rows keyed by partition, columns sparse, keys kept sorted for range scans. *Use when* writes are huge and the table can be designed around a few known query patterns.

**Graph (Neo4j):** nodes and edges. *Use when* the core queries traverse many-to-many relationships (social graph, recommendations).

**Scaling moves (apply on top of any store):**

• **Indexing** — add a secondary structure so a query stops scanning. First lever.
• **Read replicas (leader-follower)** — copy writes to followers that serve reads.

*Use when* reads dominate and slight staleness is OK.

• **Federation** — split DBs by function (users / products / forums). *Use when*

functional domains scale independently and rarely join.

• **Sharding/partitioning** — split one logical table across nodes by a shard key.

*Use when* a single node's writes or dataset are exceeded. (This skill owns it.)

• **Denormalization** — store redundant copies to skip joins. *Use when* reads

vastly outnumber writes and joins are the bottleneck.

**Polyglot persistence:** use more than one of the above, each for what it's best at (e.g. Postgres for orders, Redis for sessions, a search index for full-text). The cost is operating and reconciling several stores.

Trade-offs

| Option | What it solves | What it worsens | Change it when | |---|---|---|---| | Relational/SQL | Joins, ACID, ad-hoc queries, integrity | Single-node write ceiling; schema migrations; harder horizontal scale | Writes/size exceed one node, or schema is truly fluid → NoSQL/shard | | Document | Schema flexibility; whole-object reads | No joins; cross-document consistency is manual; query engine weaker | Data turns relational or needs multi-doc transactions → SQL | | Key-value | Extreme throughput, simple ops | Only key access; no range/secondary queries | Queries beyond the key are needed → document/wide-column | | Wide-column | Write-heavy scale, range scans on sorted keys | Must know queries up front; rigid once keyed; eventual by default | Access patterns are unknown/varied → relational | | Graph | Cheap deep relationship traversal | Niche tooling; hard to shard; weak for bulk scans | Relationships are shallow → relational/document | | Indexing | Turns scans into lookups | Slower writes; more storage; index bloat | Write amplification hurts more than the read win | | Read replicas | Offloads reads; redundancy | Replication lag → stale reads; failover/promotion logic | Stale reads unacceptable → read-from-leader / `consistency-coordination` | | Federation | Per-domain scale, smaller working sets | Cross-domain joins break; app routing logic | A single domain still won't fit → shard that domain | | Sharding | Horizontal write + storage scale | Cross-shard joins/txns hard; resharding pain; hot shards |