/deployment-gotchas

name: deployment-gotchas
description: "Use when preparing releases or deployment config — mix phx.gen.release, runtime.exs vs compile-time config, release migrations, PHX_HOST/PHX_SERVER, SSL, Dockerfile order, health checks. Not for logging/metrics (use telemetry-essentials)."

Deployment Gotchas

> Verified against Elixir 1.20, Phoenix 1.8, Ecto 3.14 — October 2026

Not a deployment guide. These are the things that break first Phoenix deploys, each tied to a real incident pattern.

RULES — Follow these with no exceptions

1. **Start from `mix phx.gen.release --docker`** — it generates `MyApp.Release`, `bin/migrate`, `bin/server` and a Dockerfile in the right build order; don't hand-write them 2. **Environment values go in `runtime.exs`; compile-time settings stay in `config/prod.exs`** — `force_ssl` and anything read with `Application.compile_env` can't be changed at boot 3. **Run migrations with `bin/migrate` before the new code takes traffic, and keep them backward-compatible** — during a rolling deploy the old version runs against the new schema (expand, then contract in a later deploy) 4. **Start the server with `bin/server` (sets `PHX_SERVER=true`) and set `PHX_HOST`** — `runtime.exs` enables the server only when `PHX_SERVER` is set; a wrong host breaks generated URLs and the websocket `check_origin` 5. **Build order is `deps.compile` → `compile` → `assets.deploy` → `release`** — colocated hooks are extracted during `mix compile`, so assets built before compiling miss them 6. **Never hardcode secrets** — read them in `runtime.exs` and fail at boot if missing 7. **Split health checks: liveness never touches the database, readiness does** — and neither returns error details (`inspect(reason)` leaks hostnames) 8. **Behind a load balancer: keep `force_ssl` with `rewrite_on: [:x_forwarded_proto]`, enable database SSL, keep `check_origin`** 9. **Use `config :logger, level: :info` in production** — `:debug` logs query parameters, which contain user data

---

1. Generate the Release Files

mix phx.gen.release --docker

It creates:

• `lib/my_app/release.ex` — `migrate/0` and `rollback/2`, runnable without Mix
• `rel/overlays/bin/migrate` — runs `./my_app eval MyApp.Release.migrate`
• `rel/overlays/bin/server` — runs `PHX_SERVER=true exec ./my_app start`
• `Dockerfile` and `.dockerignore` — a multi-stage build pinned to your Elixir/OTP versions

Hand-written `release.ex` files often use `Application.load(@app)`. That returns `{:error, {:already_loaded, _}}` when the app is already loaded. The generator uses the idempotent `Application.ensure_loaded/1`:

# lib/my_app/release.ex (generated)
defp load_app do
  # Many platforms require SSL when connecting to the database
  Application.ensure_all_started(:ssl)
  Application.ensure_loaded(@app)
end

---

2. runtime.exs vs config.exs

**The incident:** `DATABASE_URL` is set correctly in production, but the release connects to the build machine's database, or to none.

**Why:** `config.exs` and `prod.exs` are evaluated when you **build** the release and baked into it. `runtime.exs` is evaluated at **boot**.

<!-- epg:bad -->

# config/prod.exs — runs on the build machine
config :my_app, MyApp.Repo, url: System.get_env("DATABASE_URL")

# config/runtime.exs
if config_env() == :prod do
  config :my_app, MyApp.Repo,
    url: System.fetch_env!("DATABASE_URL"),
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")
end

The reverse mistake: some settings are read at compile time with `Application.compile_env/3` and can't be changed at boot. Examples are the endpoint's `force_ssl`, `code_reloader` and `debug_errors`, and the generated `:dev_routes`. Set them in `config.exs`/`prod.exs`. If `runtime.exs` sets one to a different value, the release refuses to boot ("the application :my_app has a different value set for key ... during runtime compared to compile time").

---

3. Release Migrations and Rolling Deploys

**The incident:** a deploy renames `users.name` to `users.full_name`. For the two minutes when old and new instances both serve traffic, every request on the old instances 500s.

# In production — no Mix in a release
bin/migrate
# or: bin/my_app eval "MyApp.Release.migrate()"

Run it as a pre-deploy step (a release command, init container or job) **before** new instances start. Then the old version briefly runs against the new schema, so every migration must be safe for the code that's already deployed:

| Change | Safe sequence | |---|---| | Add a column | Add it as nullable or with a default → deploy the code that writes it → backfill → add `NOT NULL` later | | Rename a column | Add the new column → write both → backfill → read the new one → drop the old one in a later deploy | | Drop a column | Stop reading it (remove it from the schema) → deploy → drop it in the next deploy | | Add an index on a big table | `create index(..., concurrently: true)` with `@disable_ddl_transaction true` and `@disable_migration_lock true` |

The exact migration code for each row (including `NOT NULL` without a full-table scan and `lock_timeout`) is in the `ecto-migrations-safety` skill.

Don't run migrations from `Application.start/2`. Every node would race to run them on boot.

---

4. PHX_SERVER, PHX_HOST and Networking

**The incident:** the release starts and the health check passes, but nothing listens on the port. Or emails link to `https://example.com`.

The generated `runtime.exs` already does the right thing:

# config/runtime.exs (generated)
if System.get_env("PHX_SERVER") do
  config :my_app, MyAppWeb.Endpoint, server: true
end

if config_env() == :prod do
  host = System.get_env("PHX_HOST") || "example.com"

  config :my_app, MyAppWeb.Endpoint,
    url: [host: host, port: 443, scheme: "https"],
    # IPv6 dual-stack: accepts IPv4 and IPv6 on all interfaces
    http: [ip: {0, 0, 0, 0, 0, 0, 0, 0}]
end

• **Start with `bin/server`**, w