/generate-confluence-scorecard

Generate Confluence Coding-Agents Scorecard

Build a Coding-Agents Scorecard Confluence page from two data sources and the canonical product→skill mapping below. Combines **test coverage** (static) with **eval scores + pass/fail** (a coder_eval run), maps both onto the org scorecard's product rows, and publishes a new Confluence page.

**Input:** `$ARGUMENTS`

• **Run dir** (required) — path to a coder_eval run, e.g. `/Users/<you>/src/coder_eval/runs/2026-05-26_04-05-02`. Must contain `run.json`.
• **`--coverage <path>`** (optional) — coverage summary path; defaults to `tests/reports/SUMMARY.md`. If stale or missing, see Phase 2.
• **`--variant <id>`** (optional) — which run variant to score; see Phase 1 nuance 5.
• **`--draft`** (optional) — create the page as a draft instead of `current`.
• **`--target <pct>`** (optional) — pass-rate bar for the gap-to-target headline; defaults to **95** (the org scorecard's stated bar). See Phase 4.
• **`--parent <pageId>`** (optional) — parent page; defaults to the org scorecard page (see Publish Target).

**Argument parsing.** The first bare (non-`--`) token is the **run dir**. Do NOT treat a second bare path as the coverage source — coverage must be passed via `--coverage` to avoid ambiguity (two raw paths are indistinguishable). If `$ARGUMENTS` is empty, ask for the run dir; do not guess the "latest" run.

**Run date** (used in the title and panels): take the run-dir folder name if it is a timestamp (`YYYY-MM-DD...`), else `run.json.start_time` (date portion). State which you used.

**Output:** A new Confluence page in space **CA** under the org scorecard, titled `Coding Agents Scorecard — Eval Scores & Test Pass/Fail (run <run-date>)`. Returns the page URL.

> This command **creates** (or, on a re-run, **updates**) a child page under the org scorecard — never overwrites the org scorecard itself. It reproduces **both** of the org card's product tables (Product Level Scorecard + Product Capability Enumeration) and fills only **derived** cells: Build/Operate/Diagnose eval scores + Tests Pass/Fail + per-tier Skills Smoke / Integration+E2E come from the run; Test Coverage / Skill Coverage come from the coverage source. It leaves Product-Coverage, CLI-* columns, Exhaustive, FDE-Golden, and Peer-Sign-off blank (not derivable here; CLI values are carried verbatim from the org card).

---

Publish Target (defaults)

| Field | Value | |---|---| | cloudId | `uipath.atlassian.net` | | spaceId (CA) | `90537164849` | | Parent page (org scorecard) | `90748321845` |

Confirm these still resolve with `getConfluenceSpaces`/`getConfluencePage` before publishing.

**Stale parent — always resolve the NEWEST dated scorecard, don't just trust the hard-coded id.** The org scorecard is re-created weekly. The hard-coded id is only a hint; a *stale-but-still-live* card (e.g. last week's) resolves fine via `getConfluencePage` and will NOT 404, so a 404-only check silently publishes the new child under an outdated parent. Instead, **always** run `searchConfluenceUsingCql` (`space=CA AND title ~ "Coding Agents Scorecard*" AND type=page`, newest first) and pick the most recent dated card (parse the `MM/DD/YYYY` in the title; the newest date wins, not merely the newest `lastModified`). Use that as the parent and tell the user which one you picked and its date. Only fall back to the hard-coded id if the search returns nothing; if that also 404s, create at the space root and flag it. The `--parent <pageId>` override always wins over this search.

Pick the matching org-card revision as your **column/row template too** (Phase 3): the scorecard's product list and the second "Product Capability Enumeration" table drift week to week (rows added/removed, e.g. Agent Hub). Read the parent you selected, not a cached layout.

**Auth / access.** If the Atlassian tools are unavailable or the CA space is not accessible, stop and report — do not write the page anywhere else.

---

Phase 1 — Parse the coder_eval run

First check `<run-dir>/run.json` exists and parses. **If missing or not valid JSON, stop** and tell the user (point at `experiment.json`/`run.md` as alternates, but do not screen-scrape `experiment.md` for per-skill numbers — `run.json` is the only structured source). Then aggregate per **skill** (derive skill from each task's `task_path`: the path segment after `tests/tasks/`).

Run this (adjust the path):

import json, re
from collections import defaultdict, Counter
d = json.load(open('<run-dir>/run.json'))
tr = d.get('task_results') or []
if not tr: raise SystemExit('run.json has no task_results — run incomplete or wrong file')

# --- keep the FULL result set for cross-cutting TAG aggregation (used below + Phase 2/3) ---
tr_all = tr                                # every task_result, ALL variants
# --- MATURITY CARRY-FORWARD: identify before any variant logic touches these rows ---
# On a `--skip-mature` night the runner does NOT re-run a task that has passed 5 runs in a row; it
# appends a synthetic SUCCESS row (`mature_skipped: true`, weighted_score 1.0, duration/cost 0) so the
# suite denominator stays the full authored size. These are REAL suite members carried forward as
# passes -- that is exactly how evalboard counts them, and the scorecard must match.
# THE TRAP: `impute_mature_skips` writes those rows WITHOUT a `variant_id` key, while every genuinely
# run row carries `variant_id: 'default'`. A naive variant filter therefore reads the carried rows as a
# phantom `None` variant and drops EVERY ONE of them -- ~45% of a nightly run (545 of 1229 rows on
# 2026-08-25) -- silently thinning every product row. That is the bug that published licensing 3/3 when
# 12 licensing tasks exist and 9 of them were carried forward green.
def matured(t): return bool(t.get('mature_skipped'))
n_mature_rows = sum(1 for t in tr if matured(t))

# --- multi-variant guard: score exactly one variant for the per-skill/dir tables ---
# Derive the variant set from RUN rows only, so c