/building-gold-corpus

name: building-gold-corpus
description: "Scaffold a synthetic gold-standard annotation project for evaluating OpenMed NER and de-identification models — label schema, annotation guidelines, BRAT or Label Studio config, and disjoint train/dev/test splits. Use when the user wants to create eval fixtures, set up annotation, define a label set, write guidelines, configure an annotation tool, or build a held-out gold set for the OpenMed eval harness. Trigger on \"gold corpus\", \"annotation project\", \"label schema\", \"annotation guidelines\", \"BRAT\", \"Label Studio\", \"train dev test split\", or \"build eval fixtures\" for OpenMed. Committed gold must be synthetic; licensed (i2b2/n2c2/MIMIC) data is eval-only and never committed."
license: Apache-2.0
metadata:
  project: OpenMed
  category: evaluation-quality
  pairs: adjacent
  version: "1.0"

Building a Gold Corpus

You can't evaluate what you can't measure against. This skill scaffolds a gold-standard annotation project whose output drops straight into the OpenMed eval harness as fixtures. The hard rule: **anything committed to the repo is synthetic**. Licensed clinical corpora (i2b2, n2c2, MIMIC) are DUA-gated — use them at eval time from the user's own copy, never check them in.

When to use this skill

• You need eval fixtures for `benchmarking-clinical-ner` or

`evaluating-with-leakage-gates` and have none.

• You're standing up an annotation effort: schema, guidelines, tool config.
• You need disciplined train/dev/test splits with no leakage between them.
• You want a small **synthetic** golden set you can commit and gate on in CI.

The OpenMed fixture shape (your target output)

Annotations must serialize to character-offset spans the harness understands:

{
  "fixtures": [
    {
      "id": "synthetic-0001",
      "language": "en",
      "text": "Ms. Jane Roe (MRN 0000000) seen 2099-01-02 for type 2 diabetes.",
      "gold_spans": [
        {"start": 4,  "end": 12, "label": "PERSON"},
        {"start": 18, "end": 25, "label": "ID_NUM"},
        {"start": 32, "end": 42, "label": "DATE"},
        {"start": 47, "end": 62, "label": "DISEASE"}
      ]
    }
  ]
}

`openmed.eval.harness.load_fixtures` accepts a top-level list or a `{"fixtures": [...]}` mapping. Offsets are character indices into `text`; labels are OpenMed-canonical.

Quick start — scaffold the project

eval/
  gold/
    guidelines.md            # annotation manual + edge-case decisions
    label_schema.json        # canonical labels + definitions + examples
    synthetic/               # COMMITTED synthetic fixtures (CI-gateable)
      train.json
      dev.json
      test.json
  external/                  # GITIGNORED: licensed DUA corpora, eval-only
    .gitignore               # *  (never commit i2b2/n2c2/MIMIC)

Verify your synthetic fixtures load and validate spans before you trust them:

from openmed.eval.harness import load_fixtures

fixtures = load_fixtures("eval/gold/synthetic/test.json")
print(len(fixtures), "fixtures;", sum(len(f.gold_spans) for f in fixtures), "spans")
# load_fixtures normalizes spans against source text and rejects duplicate ids.

Workflow

1. **Define the label schema.** Reuse OpenMed canonical labels (PERSON, DATE, ID_NUM, EMAIL, PHONE, DISEASE, DRUG, ...). Each label gets a one-line definition, in/out examples, and a boundary rule (include titles? trailing punctuation?). 2. **Write annotation guidelines.** The manual is the contract: span boundaries, nested/overlapping policy, ambiguous cases, and a decision log appended as real cases force calls. Vague guidelines → low agreement → unusable gold. 3. **Generate synthetic source text.** Compose realistic clinical narratives with *fabricated* identifiers (Faker-style names, impossible dates like 2099-, all-zero MRNs). Never paste real notes into committed data. 4. **Configure the tool.** BRAT uses `annotation.conf` (entity types) producing `.ann` standoff; Label Studio uses a labeling-config XML producing JSON. Map either back to the fixture shape above. 5. **Double-annotate and measure agreement.** Have ≥2 annotators on an overlap set; compute span-level inter-annotator agreement (F1 or Cohen's κ). Adjudicate disagreements and fold the resolutions into the decision log. 6. **Split with discipline.** Partition **by document/patient**, not by sentence, so no patient appears in two splits. Freeze test; never tune on it. 7. **Validate and commit.** Run `load_fixtures`; confirm spans align and ids are unique. Commit only the synthetic splits.

Hand-off to / from OpenMed

• **To** `benchmarking-clinical-ner`: dev/test fixtures feed `run_suite` and

`error_report` for the NER scorecard.

• **To** `evaluating-with-leakage-gates` and `gating-deid-leakage`: the synthetic

held-out set is exactly what the release gates and the CI gate run against.

• **To** `building-with-openmed`: synthetic notes can be generated by running

surrogate replacement through `openmed.deidentify(method="replace")`.

• **Pairs with** `auditing-subgroup-fairness`: tag each gold span with a `group`

in `metadata` so `fairness_report` can slice by demographic surrogate.

Edge cases & gotchas

• **Committed = synthetic. No exceptions.** Real PHI in the repo is a breach even

if the repo is private. Generate identifiers; don't transcribe them.

• **DUA data is eval-only.** Load i2b2/n2c2/MIMIC from `eval/external/` (gitignored)

at runtime under the user's license; results may be reported, data never shared.

• **Split by patient, not by line.** Sentence-level splitting leaks a patient's

style/identifiers across train and test and inflates scores.

• **Offsets must be character indices into *this* `text`.** Re-tokenization or

whitespace edits silently shift offsets; re-validate with `load_fixtures`.

• **Label the fairness surrogate, not real demographics.** Put a synthetic

`group` tag in span `metadata`; don't store re