/caching

name: caching
description: Use when adding or debugging a cache. Covers cache placement, invalidation strategies, stampede protection, TTL selection, and the consistency you are trading away.
metadata:
  category: backend
  version: 1.0.0
  tags: [caching, redis, invalidation, performance, stampede]

Caching

Purpose

Add a cache with a clear invalidation story, or find out why the existing one is serving stale data. A cache is a second source of truth; introducing one is a consistency decision, not just a performance one.

When to Use

• A read path is slow and the data is read far more often than it is written.
• An upstream dependency is expensive, rate-limited, or unreliable.
• Debugging stale reads, cache stampedes, or unexplained memory growth.

Capabilities

• Cache placement: client, CDN, application, database.
• Strategies: cache-aside, read-through, write-through, write-behind.
• Invalidation: TTL, explicit, event-driven, versioned keys.
• Stampede protection: locking, early recomputation, request coalescing.
• Negative caching and hot-key mitigation.

Inputs

• The read/write ratio and the cost of the uncached path.
• How stale the data is allowed to be — in seconds, agreed with the product owner.
• Cardinality of the key space and expected memory footprint.

Outputs

• A key schema, a TTL, and an invalidation trigger for each cached entity.
• Stampede protection on any key expensive enough to matter.
• A hit-rate metric, because a cache you do not measure is a cache you do not understand.

Workflow

1. **Prove the need** — Measure the uncached path first. A cache in front of a missing index is a permanent workaround for a five-minute fix. 2. **Decide the staleness budget** — Ask what breaks if a user sees data five seconds old. The answer determines the TTL and whether you need explicit invalidation. 3. **Design the key** — Include everything that varies the result: tenant, locale, permission scope. A key that omits a dimension leaks data between users. 4. **Choose invalidation** — TTL alone for tolerant data. TTL plus explicit deletion on write for data that must be fresh. Versioned keys when deletion is unreliable. 5. **Protect against stampedes** — When a hot key expires, every concurrent request misses simultaneously and hits the origin at once. Use a lock or single-flight. 6. **Measure** — Hit rate, latency at each layer, and eviction rate. Falling hit rate with rising memory means the key space is too large.

Best Practices

• Cache the expensive computation, not the whole response. Response-level caching multiplies the key space by every variant.
• A cache key missing the tenant or the user's permission scope is a data-leak vulnerability, and it will be found by a customer.
• Never cache with an unlimited TTL and rely on invalidation alone. Invalidation fails; the TTL is the safety net.
• Set memory limits and an eviction policy explicitly (`allkeys-lru`). A Redis instance with no `maxmemory` will be killed by the OOM killer.
• Do not cache errors indefinitely, but do cache "not found" briefly — otherwise a bad ID becomes an origin DoS.
• Warm caches before cutting traffic over. A cold cache under full load is an outage.

Examples

**Single-flight cache-aside, preventing a stampede:**

async def get_pricing(tenant_id: str, sku: str) -> Pricing:
    key = f"pricing:v3:{tenant_id}:{sku}"

    if (cached := await redis.get(key)) is not None:
        return Pricing.model_validate_json(cached)

    # Only one caller per key computes; the rest wait for the result.
    lock_key = f"{key}:lock"
    if await redis.set(lock_key, "1", nx=True, ex=10):
        try:
            pricing = await compute_pricing(tenant_id, sku)      # expensive
            await redis.set(key, pricing.model_dump_json(), ex=300)
            return pricing
        finally:
            await redis.delete(lock_key)

    # Lost the race: wait briefly for the winner, then fall back to computing.
    for _ in range(20):
        await asyncio.sleep(0.05)
        if (cached := await redis.get(key)) is not None:
            return Pricing.model_validate_json(cached)

    return await compute_pricing(tenant_id, sku)

Notes

• The `v3` in the key is a schema version. When the cached shape changes, bump it — this invalidates the whole namespace atomically without a scan-and-delete.
• Adding jitter to TTLs (`ex=300 + random.randint(0, 60)`) prevents a cohort of keys written together from expiring together.
• A 95% hit rate sounds excellent, but if the 5% of misses are all on the single hottest key, the origin still sees the full load. Look at misses per key, not just the aggregate.