/annotate

name: annotate
description: Build and verify a PII gold set with HUMAN annotators (first-class). Launch the browser annotator, label spans per the codebook, export per-annotator label files, then compute inter-annotator agreement (Cohen's/Fleiss' kappa) and draft an adjudicated gold. Use when the user says "annotate PII", "label this transcript", "build a gold set", "inter-annotator agreement", "review annotations", "adjudicate labels", or wants to measure/defend a de-identification gold standard. Local-only: synthetic or consented data only; annotators' names and transcript text stay on the machine — only labels/stats are collected, nothing PII is re-shared.

confide:annotate — human PII gold set + inter-annotator agreement

Humans label PII spans in a transcript; you measure how much they agree (κ) and draft an adjudicated gold from their labels. Annotators are first-class here — most of this skill is plain instructions FOR a person doing the labelling, plus a coordinator path to score it.

Privacy invariants (do not violate)

• **Synthetic or consented data only.** Never load a real client transcript the person did not

consent to share. When in doubt, anonymize first (`confide:anon`) and annotate the GREEN copy.

• **Names stay local.** The annotator's labels (which contain real surface text spans) live in

their browser and the exported JSON file on their own machine. Collect label files locally.

• **Nothing PII is re-shared.** Only κ / F1 / disagreement *clusters* travel between people if

needed. The transcript text and the original PII are never re-distributed by this skill.

Bundled assets

• `assets/annotator.html` — zero-install browser annotation tool (EN/RU, runs offline).
• `references/codebook.md` — the labelling rulebook (10 PII types, direct/quasi, harm).
• `references/tool-guide.md` — how to drive the tool + scorer step by step.
• `scripts/score_iaa.py` — Cohen's/Fleiss' κ, span-F1, disagreement queue, draft gold (stdlib).
• `scripts/gold_to_labels.py` — turn an existing gold into a "reference annotator" to test solo.

---

FOR THE ANNOTATOR (no coding needed)

1. **Open the tool.** Double-click `assets/annotator.html` (or open it in Chrome/Firefox/ Safari). It runs entirely in your browser — nothing is uploaded; labels stay on your machine until you Export. 2. **Read the rules.** Open `references/codebook.md` first. It defines the 10 types (PERSON, LOCATION, ORG, PHONE, EMAIL, ID, DATE, MEDICATION, AGE, PROFESSION), what counts as a span (the *minimal* identifying text), and direct vs. quasi-identifier. 3. **Set your annotator id and load the transcript** in the tool (e.g. `A`, `B`, or your name). Use only synthetic or consented text. 4. **Label every PII span.** Select the minimal text that identifies a real person (the client or third parties they mention) and assign its type. Record direct/quasi, entity id, role, and harm as the codebook describes. **Do not rewrite or redact — only label.** 5. **When unsure, log it — don't guess silently.** Add a note starting with `QUESTION:` on the span (e.g. `QUESTION: gym or city?`). These flow straight into the adjudication queue. 6. **Export.** Click Export → you get `labels.<doc>.<annotator>.json` (schema: `{doc_id, annotator, text, spans:[{start,end,text,type,...}]}`). Keep it local and hand only this file to the coordinator. Two+ people should label the *same* doc independently (blind) for a meaningful κ.

FOR THE COORDINATOR (measure + adjudicate)

1. **Collect** every `labels.<doc>.<annotator>.json` into one folder, e.g. `labels/`. 2. **Score IAA:**

   python3 skills/annotate/scripts/score_iaa.py --labels-dir labels/ --out-dir results/

It writes (per doc + overall): **Cohen's κ** (pairwise), **Fleiss' κ** (3+ annotators), **span-F1**, a **disagreement queue** (`*-iaa-disagreements.json`: every cluster annotators don't fully agree on, plus any `QUESTION:` spans), and a **draft adjudicated gold** (`*-adjudicated-gold-draft.json`: majority span per overlap-cluster, ties/questions marked `needs_review:true`). Character-level κ sidesteps tokenization disputes. 3. **Target κ ≥ 0.80** = a defensible gold. Lower usually means an unclear codebook rule, not a careless annotator — fix the rule and re-label, don't just discard. 4. **Adjudicate.** Walk the disagreement queue with a human adjudicator; resolve each `needs_review` cluster. The resulting label set is the published gold; report post-adjudication κ too. Nothing is ever auto-finalised.

Test the loop solo (no second person yet)

Treat an existing gold JSONL as one "reference annotator", label the same doc yourself in `annotator.html` as another, then score the pair:

python3 skills/annotate/scripts/gold_to_labels.py --gold GOLD.jsonl --name gold --out-dir labels/
# label the same doc yourself in annotator.html as "me" -> drop labels.<doc>.me.json into labels/
python3 skills/annotate/scripts/score_iaa.py --labels-dir labels/ --out-dir results/

(`--sessions-dir DIR` lets `gold_to_labels.py` read transcript text from disk so char offsets match the gold exactly.)

Output

IAA results (κ, F1) + a disagreement list + a draft adjudicated gold — labels/stats only. Transcript text and original PII stay local; only what's needed to adjudicate is shared.