/authoring-model-cards

name: authoring-model-cards
description: "Generate a model card for an OpenMed clinical NER or de-identification model documenting intended use, quantitative metrics, subgroup performance, limitations, and a medical-device disclaimer for clinical AI governance. Use when the user wants to write or update a model card, a README model section, or governance documentation, or to turn OpenMed eval outputs (release gate report, fairness_report, error_report) into the card's metrics and limitations sections. Trigger on \"model card\", \"intended use\", \"model documentation\", \"governance\", \"limitations section\", \"datasheet\", or \"FDA/ONC transparency\" for an OpenMed model."
license: Apache-2.0
metadata:
  project: OpenMed
  category: evaluation-quality
  pairs: after
  version: "1.0"

Authoring Model Cards

A model card is the honest spec sheet for a model: what it's for, how well it works, where it breaks, and who it might fail. For clinical models this is governance-critical — an undocumented de-id model is one nobody can sign off on. This skill fills a model card directly from OpenMed eval outputs so the numbers are reproducible, not aspirational.

When to use this skill

• You're publishing or updating an OpenMed model and need its card.
• You have eval artifacts (`GateReport`, `fairness_report`, `error_report`) and

need to turn them into intended-use, metrics, and limitations sections.

• A clinical AI governance / model-risk review needs a transparency document.

Run the evals **first** (see `evaluating-with-leakage-gates`, `benchmarking-clinical-ner`, `auditing-subgroup-fairness`); this skill documents their results — it does not generate the numbers.

Card sections (Mitchell et al., + clinical extensions)

See `references/model-card-sections.md` for the full section-to-source map. The load-bearing sections for an OpenMed model:

• **Model details** — repo id, family, tier, format, params, milestone, license

(Apache-2.0). Pull from the `GateReport` identity fields.

• **Intended use** — the clinical task and the deployment envelope.
• **Out-of-scope / misuse** — explicitly: not a medical device; not for autonomous

clinical decisions; de-id is verified, not assumed.

• **Metrics** — entity-level P/R/F1 and, for de-id, residual leakage + per-label

recall floors and the gate decision.

• **Quantitative analysis (subgroups)** — per-group leakage/recall from

`fairness_report`, including which groups lack data.

• **Limitations** — error patterns from `error_report`; calibration assumptions.
• **Caveats & disclaimer** — the medical-device disclaimer.

Quick start — fill the card from eval outputs

from openmed.eval import (
    run_suite, ReleaseGate, fairness_report, error_report,
)

report = run_suite("eval/gold/test.json", suite="golden",
                   model_name="OpenMed/Privacy-PII-Detection", device="cpu",
                   metadata={"family": "PII", "tier": "base",
                             "policy": "hipaa_safe_harbor"})

gate = ReleaseGate(milestone="v1.6", policy="hipaa_safe_harbor").evaluate(report)
fair = fairness_report("OpenMed/Privacy-PII-Detection", "golden")
errs = error_report("OpenMed/Privacy-PII-Detection", "eval/gold/test.json")

card = {
    "model_details": {
        "repo_id": gate.repo_id, "family": gate.family, "tier": gate.tier,
        "format": gate.format, "license": "Apache-2.0",
    },
    "metrics": {
        "exact_span_f1": report.metrics["exact_span_f1"]["f1"],
        "residual_leakage_rate": gate.residual_leakage_rate,
        "critical_leakage_count": gate.critical_leakage_count,
        "per_label_recall": dict(gate.per_label_recall),
        "release_decision": gate.decision,            # RELEASABLE / QUARANTINED
    },
    "subgroup_analysis": fair.to_dict(),              # per-group leakage/recall
    "limitations": errs.to_dict()["confusion_matrix"],
}
# Render `card` into Markdown front matter + body (or the HF card template).

`error_report` and `fairness_report` carry no plaintext PHI (offsets + hashes), so their output is safe to paste into a public card.

Workflow

1. **Gather artifacts.** Gate report, fairness report, error report — all from a pinned model + synthetic eval set. 2. **Fill model details** from the `GateReport` identity fields so the card, `models.jsonl`, and the README cannot drift (the gate's `manifest_coherence` and `model_card` checks enforce this). 3. **Write intended use narrowly.** Name the clinical task, language(s), and the deployment envelope. Over-broad intended-use is the most common card failure. 4. **State out-of-scope and the disclaimer** plainly (see template below). 5. **Report metrics with their floors.** For de-id, lead with leakage and the gate decision, not F1. 6. **Report subgroups honestly**, including the documentation gap: if race/ ethnicity isn't available, say so rather than implying parity. 7. **List limitations from real errors**, not boilerplate — cite the confusion matrix's worst cells.

Disclaimer block (paste & adapt)

> This model assists clinical text processing and is **not a medical device**. > It does not make autonomous clinical decisions. De-identification output must be > independently verified before any data is shared; residual PHI risk is never > zero. Validate on your own population before deployment.

Hand-off to / from OpenMed

• **From** `evaluating-with-leakage-gates` (`GateReport`),

`benchmarking-clinical-ner` (`error_report`), and `auditing-subgroup-fairness` (`fairness_report`): these are the card's evidence.

• **To** `building-with-openmed` / `models.jsonl`: keep card front matter

(license, task, languages) coherent with the manifest — the gate checks it.

• **Pairs with** `gating-deid-leakage`: cite the green gate as the card's

release evidence.

Edge cases & gotchas

• **Don't claim numbers you can't reproduce.** Every metric in the card should

trace to an eval artifact and a pinn