/blob-store

name: blob-store
description: This skill should be used when the user wants a "blob store" or "object storage", names "S3" or an S3-compatible store, needs to "store images / video / files", asks about "multipart upload" or "resumable upload", "signed / presigned URLs", "media storage", "unstructured data at scale", object "versioning", storage "tiering" (hot/cold/archive), or "erasure coding" vs replication for durability. Use it whenever a design must hold large unstructured objects (photos, video, backups, logs, ML datasets) and serve them cheaply and durably, even if the user just says "where do we put the files".

Blob store

Store large, immutable, unstructured objects — images, video, backups, model weights, document blobs — in a flat namespace keyed by a string, replicated for durability and served by direct download. Getting it wrong means stuffing multi-megabyte blobs into a row-oriented database (where they bloat the working set, wreck cache locality, and cap throughput) or hand-rolling a file server that loses data on the first disk failure.

When to reach for this

Objects are large (KB to GB), written once and read many times, and you only ever fetch them whole by key — never query *inside* them. Photo/video stores, user uploads, backups, data-lake/ML datasets, static-site assets, log archives. The access pattern is `PUT key → GET key`, durability matters, and the total volume is too large or too cold to sit in a primary database.

When NOT to

Small structured records you query, filter, sort, or join — that is `data-storage`. Data that needs transactions, secondary indexes, or partial updates (blobs are replace-whole, not edit-in-place). Low-latency reads of tiny values (a KV cache or `caching` wins). A few files on one box that never grow — the local filesystem is fine; a blob store is operational overhead you do not need yet (YAGNI). Naming "object storage" for a workload that is really a database is failure mode #2.

Clarify first

• **Object size distribution** — average and p99 size? (Decides chunking, multipart

thresholds, and whether reads stream or buffer.) → `back-of-the-envelope`.

• **Read:write ratio and access recency** — write-once/read-many? How fast does data

go cold? (Drives tiering and CDN fronting.)

• **Durability and availability target** — how many nines of durability? Can a read

briefly fail or must it always succeed? (Replication vs erasure coding, multi-region.)

• **Access control** — public, private, or time-limited per-object grants? (Signed URLs.)
• **Mutability and history** — do objects change? Must old versions be retained

(compliance, undo)? (Versioning + lifecycle.)

• **Egress profile** — who reads, from where, how often? (CDN offload, egress cost.)

The options

**Durability scheme** (how many copies, what shape)

• **N-way replication** — store N full copies on different nodes/racks/AZs. Use when

objects are small, hot, and latency matters; simplest to reason about.

• **Erasure coding (EC)** — split an object into *k* data + *m* parity shards; any *k*

reconstruct it. Use for large/cold data at scale — same durability as replication at ~1.4x overhead instead of 3x. (Mechanics in `references/deep-dive.md`.)

**Storage tier** (price/latency/retrieval trade)

• **Hot/standard** — millisecond reads, highest $/GB. Use for actively served objects.
• **Cool/infrequent** — cheaper storage, retrieval fee/slightly higher latency. Use

for backups and data read a few times a month.

• **Archive/cold** — cheapest storage, minutes-to-hours retrieval. Use for compliance

retention and rarely-touched data; never for anything on a request path.

**Upload path**

• **Single PUT** — one request. Use for small objects (under the multipart threshold).
• **Multipart / resumable** — split into parts, upload in parallel, retry per-part,

commit on completion. Use for large objects and flaky networks; the default above the threshold.

**Mutation model**

• **Immutable + versioning** — each write is a new version; deletes are tombstones.

Use when history, undo, or accidental-overwrite protection matters.

• **Overwrite-in-place (last-writer-wins)** — simplest; no history. Use when only the

latest object matters and storage of old copies is waste.

Trade-offs

| Option | What it solves | What it worsens | Change it when | |---|---|---|---| | N-way replication | Simple, fast reads, fast rebuild | 3x+ storage cost | Data is large/cold and cost dominates → erasure coding | | Erasure coding | Same durability at ~1.4x storage | CPU + multi-node read on every fetch; slow small-object reads; costly rebuild | Objects are small/hot and latency matters → replication | | Hot tier | Low-latency serving | Highest $/GB | Data goes cold and is rarely read → cool/archive | | Archive tier | Cheapest at-rest storage | Minutes–hours to first byte; retrieval fees | Anything ends up on a latency-sensitive path → hot/cool | | Multipart/resumable upload | Large files survive flaky links; parallel throughput | More client logic; orphaned parts cost money | Objects are small → single PUT | | Versioning | Undo, history, overwrite protection | Storage grows silently; needs lifecycle expiry | Only latest matters → overwrite, last-writer-wins | | Signed URLs | Offload transfer off your app; scoped access | Leaked/over-broad URLs; clock-skew expiry bugs | Content is fully public → CDN + public read |

Behavior under stress

A blob store rarely "falls over" the way a database does, but it amplifies trouble in specific ways.

• **Hot object / hot prefix:** a viral file or a key scheme where many writes share a

prefix concentrates load on one partition. *Mitigate:* front hot reads with a CDN (`content-delivery`), randomize/hash key prefixes, replicate the hot object.

• **Metadata-index bottleneck:** the index that maps key → shard locations is the real

SPOF and the throughput ceiling (millions of tiny objects hurt far more than a few huge ones). *Mitig