/otp-thinking

name: otp-thinking
description: This skill should be used when the user asks to "add background processing", "cache this data", "run this async", "handle concurrent requests", "manage state across requests", "process jobs from a queue", "this GenServer is slow", or mentions GenServer, Supervisor, Agent, Task, Registry, DynamicSupervisor, handle_call, handle_cast, supervision trees, fault tolerance, "let it crash", or choosing between Broadway and Oban.

OTP Thinking

Paradigm shifts for OTP design. These insights challenge typical concurrency and state management patterns.

The Iron Law

GENSERVER IS A BOTTLENECK BY DESIGN

A GenServer processes ONE message at a time. Before creating one, ask: 1. Do I actually need serialized access? 2. Will this become a throughput bottleneck? 3. Can reads bypass the GenServer via ETS?

**The ETS pattern:** GenServer owns ETS table, writes serialize through GenServer, reads bypass it entirely with `:read_concurrency`.

**No exceptions:** Don't wrap stateless functions in GenServer. Don't create GenServer "for organization".

GenServer Patterns

| Function | Use For | |----------|---------| | `call/3` | Synchronous requests expecting replies | | `cast/2` | Fire-and-forget messages |

**When in doubt, use `call`** to ensure back-pressure. Set appropriate timeouts for `call/3`.

Use `handle_continue/2` for post-init work—keeps `init/1` fast and non-blocking.

Task.Supervisor, Not Task.async

`Task.async` spawns a **linked** process—if task crashes, caller crashes too.

| Pattern | On task crash | |---------|---------------| | `Task.async/1` | Caller crashes (linked, unsupervised) | | `Task.Supervisor.async/2` | Caller crashes (linked, supervised) | | `Task.Supervisor.async_nolink/2` | Caller survives, can handle error |

**Use Task.Supervisor for:** Production code, graceful shutdown, observability, `async_nolink`. **Use Task.async for:** Quick experiments, scripts, when crash-together is acceptable.

DynamicSupervisor + Registry = Named Dynamic Processes

DynamicSupervisor only supports `:one_for_one` (dynamic children have no ordering). Use Registry for names—never create atoms dynamically:

defp via_tuple(id), do: {:via, Registry, {MyApp.Registry, id}}

**PartitionSupervisor** scales DynamicSupervisor for millions of children.

:pg for Distributed, Registry for Local

| Tool | Scope | Use Case | |------|-------|----------| | Registry | Single node | Named dynamic processes | | :pg | Cluster-wide | Process groups, pub/sub |

`:pg` replaced deprecated `:pg2`. **Horde** provides distributed supervisor/registry with CRDTs.

Broadway vs Oban: Different Problems

| Tool | Use For | |------|---------| | Broadway | External queues (SQS, Kafka, RabbitMQ) — data ingestion with batching | | Oban | Background jobs with database persistence |

Broadway is NOT a job queue.

Broadway Gotchas

**Processors are for runtime, not code organization.** Dispatch to modules in `handle_message`, don't add processors for different message types.

**one_for_all is for Broadway bugs, not your code.** Your `handle_message` errors are caught and result in failed messages, not supervisor restarts.

**Handle expected failures in the producer** (connection loss, rate limits). Reserve max_restarts for unexpected bugs.

Supervision Strategies Encode Dependencies

| Strategy | Children Relationship | |----------|----------------------| | :one_for_one | Independent | | :one_for_all | Interdependent (all restart) | | :rest_for_one | Sequential dependency |

Use `:max_restarts` and `:max_seconds` to prevent restart loops.

Think about failure cascades BEFORE coding.

Abstraction Decision Tree

Need state?
├── No → Plain function
└── Yes → Complex behavior?
    ├── No → Agent
    └── Yes → Supervision?
        ├── No → spawn_link
        └── Yes → Request/response?
            ├── No → Task.Supervisor
            └── Yes → Explicit states?
                ├── No → GenServer
                └── Yes → GenStateMachine

Storage Options

| Need | Use | |------|-----| | Memory cache | ETS (`:read_concurrency` for reads) | | Static config | :persistent_term (faster than ETS) | | Disk persistence | DETS (2GB limit) | | Transactions/Distribution | Mnesia |

:sys Debugs ANY OTP Process

:sys.get_state(pid)        # Current state
:sys.trace(pid, true)      # Trace events (TURN OFF when done!)

Telemetry Is Built Into Everything

Phoenix, Ecto, and most libraries emit telemetry events. Attach handlers:

:telemetry.attach("my-handler", [:phoenix, :endpoint, :stop], &handle/4, nil)

Use `Telemetry.Metrics` + reporters (StatsD, Prometheus, LiveDashboard).

Red Flags - STOP and Reconsider

• GenServer wrapping stateless computation
• Task.async in production when you need error handling
• Creating atoms dynamically for process names
• Single GenServer becoming throughput bottleneck
• Using Broadway for background jobs (use Oban)
• Using Oban for external queue consumption (use Broadway)
• No supervision strategy reasoning

**Any of these? Re-read The Iron Law and use the Abstraction Decision Tree.**