/by-failure-diagnosis

id: "skill_21cae38a6ba64f28a898ac1c9e2cecff"
name: "by-failure-diagnosis"
display-name: "BY Failure Diagnosis"
short-description: "Statistical post-mortem for low-yield design campaigns — runs Mann-Whitney U tests across continuous features (ipSAE, ipTM, pLDDT, RMSD, liabilities, CDR3 length, hydrophobic fraction, net charge) to identify which metrics most strongly discriminate PASS from FAIL. Use when a campaign pass rate drops below 20% and at least 30 designs have been scored."
category: "diagnosis"
keywords: "failure diagnosis, mann-whitney, statistical test, pass rate, discriminating features, threshold tuning, campaign post-mortem, effect size, BH correction, design feedback loop"
version: "1.0"
last-updated: "2026-05-20"
mcp_tools: ["mcp__by-screening__screen_diagnose_failures"]

BY Failure Diagnosis Skill

Closing the design feedback loop requires understanding **why** designs fail, not just **that** they fail. This skill compares the distribution of every continuous feature between PASS and FAIL designs using non-parametric statistics, ranks features by discriminating power, and translates the result into concrete threshold or campaign-parameter changes for the next iteration.

It is the bridge between **screening** (which produces PASS/FAIL labels) and **campaign optimization** (which adjusts parameters for the next round).

---

When to Use This Skill

Use this skill when:

• ✅ **Pass rate is below 20%** in a screening round and you need to know why before re-spending compute
• ✅ **At least 30 designs have been scored** with a `status` field (`PASS` or `FAIL`)
• ✅ **You have numeric features per design** (ipSAE, ipTM, pLDDT, RMSD, liabilities, net_charge, hydrophobic_fraction, cdr3_length)
• ✅ **User explicitly asks** "why are my designs failing?", "diagnose failures", "what's going wrong?"
• ✅ **Before the active-learning step** in a multi-round campaign (route diagnosis → optimizer)
• ✅ **After a screening regression** where pass rate dropped versus a prior round

Don't use this skill when:

• ❌ **Fewer than 30 designs total** — statistical power is too low; the test will be noisy. Score more designs first.
• ❌ **No FAIL designs (100% pass rate)** — there is nothing to compare against. Either tighten thresholds or move to lab submission.
• ❌ **No PASS designs (0% pass rate)** — there is nothing to compare against. Use **by-hypothesis-debate** to pick a new strategy before spending more compute.
• ❌ **You want to redesign individual residues** — that is per-design rationale, not population statistics. Use **by-epitope-analysis** instead.
• ❌ **You want to predict structures or score new designs** — use **protenix** or **by-scoring** instead.
• ❌ **The campaign has different scoring criteria across rounds** — comparing apples to oranges; run diagnosis within a single round only.

---

Quick Start

The most common invocation: run the MCP tool against a campaign's screening output.

import json

# Load campaign screening results (one dict per design)
with open("campaigns/<target>/<campaign_id>/screening_results.json") as fh:
    designs = json.load(fh)

# Call the MCP tool with the array serialized as JSON
result = mcp__by-screening__screen_diagnose_failures(
    scores_json=json.dumps(designs),
    pass_key="status",
    pass_value="PASS",
)

# Parse and present
diag = json.loads(result)
print(diag["formatted"])           # human-readable table
print(diag["recommendations"])     # top-3 actionable suggestions

✅ **VERIFICATION:** Expect output like `Failure Diagnosis (12/80 passed, 15% rate)` followed by a table sorted by p-value, with effect sizes and recommendations.

For ad-hoc CSV-based diagnosis (no MCP server required), use the included scripts:

python3 scripts/diagnose_from_csv.py --input designs.csv --status-col status
python3 scripts/plot_distributions.py --input designs.csv --output diagnostics.pdf --top-n 5

---

Installation

| Software | Version | License | Commercial Use | Installation | |----------|---------|---------|----------------|--------------| | Python | ≥3.10 | PSF | ✅ Permitted | system | | scipy | ≥1.11 | BSD-3 | ✅ Permitted | `pip install scipy` | | numpy | ≥1.24 | BSD-3 | ✅ Permitted | `pip install numpy` | | pandas | ≥2.0 | BSD-3 | ✅ Permitted | `pip install pandas` | | matplotlib | ≥3.7 | PSF-based | ✅ Permitted | `pip install matplotlib` (only for `plot_distributions.py`) | | proteus_cli | bundled | proprietary | internal | included with BY |

**License Compliance:** All third-party packages permit commercial use in AI applications.

**System requirements:** No GPU needed. Diagnosis runs CPU-only in seconds for up to 10,000 designs. No internet required.

---

Inputs

**Required:**

• **Designs array (or CSV)** with one row per design:
• A status column (default name `status`) with values `PASS` or `FAIL` (uppercase)
• At least one numeric feature column from this set: `ipsae`, `ipsae_min`, `iptm`, `plddt`, `rmsd`, `net_charge`, `hydrophobic_fraction`, `liabilities`, `cdr3_length`
• Minimum 3 designs in each of PASS and FAIL groups (statistical floor)
• Recommended minimum 30 designs total (≥10 in each group) for usable power

**Alternative inputs:**

• **JSON array** (one object per design) — pass as `scores_json` to the MCP tool
• **pandas DataFrame** — convert to records with `df.to_dict(orient="records")` before passing
• **CSV file** — use `scripts/diagnose_from_csv.py` directly

**Optional:**

• **`pass_key`** — column name for status (default `status`)
• **`pass_value`** — value indicating PASS (default `PASS`)
• **Extra feature columns** — the MCP tool only tests the canonical 9 features; custom features require the CSV script (see `scripts/diagnose_from_csv.py --help`)

See [references/failure-patterns.md](references/failure-patterns.md) for which features matter for which failure modes.

---

Outputs

**Primary results (JSON object from MCP tool):**

• `total_designs`, `passed`, `failed