handoff-2026-07-27

Coordinator handoff — 2026-07-27

Supersedes `handoff-2026-07-26.md`. Read this before `aw work ready`.

**Every state claim below was measured at the time of writing. Re-measure before acting — most of these moved several times in one night.**

What Juan said, in his words

> "we did not think enough when building aw about the need to actually deploy > it. that's the learning."

> "we do not explicitly reject anything that current clients are running."

> "shippability (the ability to release without breaking anything in the field) > should be paramount in your mind."

That is the frame. Everything below is downstream of it.

The rule that broke us, and the rule that replaced it

`.claude/skills/cross-repo-change/SKILL.md` said *"Do NOT accept both old and new formats during transition when both sides deploy atomically. Pick one format."* Correct for aweb↔aweb-cloud, which ship in one Docker image. **Catastrophic for published clients, which never deploy atomically with anything** — a running agent keeps its loaded module until it *restarts*.

A chat mark-read change made `message_ids` required and explicitly rejected `up_to_message_id`. Every published client sent the old field. Deploying it would have broken every client in the world. **The process was not ignored — it was followed.**

Fixed at `e1352042`: the skill now opens with a shippability section — a released client is a permanent constraint; additive by default; narrowing requires naming every consumer and proving each deploys atomically; removal is a separate change gated on *measuring* nobody sends the old shape. The old principle and anti-pattern are now explicitly scoped to the cloud boundary with "do not generalise it."

Shipped and verified against the live artifact

| artifact | version | verified how | |---|---|---| | AC hosted | **v0.7.8**, `git_sha 1f291a72` | `app.aweb.ai/health` build block | | aweb pin in prod | **1.26.24** (held deliberately) | same | | aweb server | **1.26.29** on PyPI | registry, not the workflow | | channel | **1.7.0** on npm + marketplace | published tarball read directly | | **aw CLI** | **1.34.0** on npm | published binary run against **prod 1.26.24**: exit 0 | | channel 1.5.2 | deprecated | raw registry, cache-busted |

v0.7.8 carries the P0 security backlog that had been merged-but-undeployed: `aajh` (forged XFF → rate-limit bypass), `aakf` (request/response **bodies** in prod logs), `aamn` (sender timestamp ordering an inbox), `aajk`, plus billing correctness.

The tmux picture is bigger than one server — I got this wrong

**There are TWO live tmux servers**, and an inherited `TMUX` env var **overrides `TMUX_TMPDIR`**. Every agent in a pane has `TMUX` set, so the unsafe path is the default path.

/tmp/tmux-501/default 7 sessions, ~40 windows — THE FLEET ac(14) atext(9) tsm(7) main(4) pi-agents(3) aweb-oas(2) oas-test(1) ~/.aweb/tmux/cli/tmux-501/default the coordinator's own session

I ran `tmux ls` plus four `TMUX_TMPDIR` variants, got identical output every time, and concluded there was one server. My inherited `TMUX` won every time — **the server holding the entire fleet was invisible to every tmux command I ran that night.** A leaked `kill-server` on the default socket takes down ac, atext, oas, tsm and pi-agents at once, which is the event `CLAUDE.md` records twice.

So: **`tmux list-sessions` is not verification.** Enumerate sockets and probe each with `tmux -S <socket> list-sessions` — 19 existed, 2 responded. I removed the 17 dead ones (re-probing each immediately before deleting), so a sweep now sees 2 real sockets instead of 19.

In flight

**aweb 1.26.29** — carries `aapc`, merged at `93dd3134`. ac-operations is running the gate. This is the only thing between us and a PyPI `latest` whose migration works.

The original incident, in one paragraph

Channel 1.5.2 shipped with `mailAcknowledgment: "manual"` — it presented mail and never marked it read, so the server re-announced the backlog on every connect. One option flip. Fixed in 1.6.0, published 2026-07-25 — and **two days later 25 of 26 processes were still running 1.5.2**, because a running process keeps its loaded module until restart. The fix was correct, published, and irrelevant. Everything else that night was diagnosis being confidently wrong.

Open, with honest status

• **`aapb` P0 — memory leak.** ~50–60 MB/hour, 2 GB ceiling every ~30h, then

restarts that kill every SSE stream. **Per-client**: ~5–9 MB/day/agent, so it scales with fleet size — a *scaling ceiling*, not a regression. **A deploy resets it and buys ~30 quiet hours; quiet is not a fix.** Juan deprioritised it: our own agents are the fleet. Ruled out: chat pubsub (closed in a `finally`), the database.

• **`aang` P1** — retitled. Stream flapping is *solved* (server restarts from

`aapb`). The real defect is that **a notification can silently fail to reach an agent for an unbounded period with no signal at either end** — measured at **21h51m on three P0 assignments**, and the resend didn't rescue it either.

• **`aapd` P2** — `chat_read_receipts` has no FK to `chat_participants` and

participants are hard-deleted (`left_at` set on 0 of 2,436 rows). Root cause of the orphans; `aapc` only makes the backfill tolerate them.

• **`aaoo` P3** — event stream resolves credential DIDs ∪ agent-row DIDs while

inbox and ack resolve credential DIDs only. Real, not firing. **Do not "fix" by narrowing the stream** — that silently stops notifying agents whose row DIDs legitimately differ.

• **`aaou` P2** — `register_tools`'s `hosted_encryptor`/`hosted_decryptor`

parameter *names* are consumed by AC via `inspect.signature`. Renaming them silently disables hosted encryption rather than raising.

• **17 processes still on 1.5.2.** Restarting picks up 1.7.0, fixes unacked mail

*and* moves them off the clobbered gl