/artifact-detection

name: artifact-detection
description: Detect annotation artifacts and shortcuts in benchmarks
execution: tactic

Artifact Detection Tactic

Systematically probe benchmarks for annotation artifacts, dataset shortcuts, and spurious correlations that allow models to achieve high scores without the intended capability.

Stages

Stage 1: Hypothesis-Only Baseline Test

Search literature for evidence that partial-input baselines achieve unexpectedly high performance:

• Hypothesis-only baselines (NLI without premise)
• Question-only baselines (QA without context)
• Label-word frequency baselines
• Majority-class and surface-pattern baselines

**Search queries**: "[benchmark] annotation artifacts", "[benchmark] hypothesis only", "[benchmark] spurious correlations", "[benchmark] dataset bias"

If published partial-input results exist, record performance gap between partial and full input. Gap < 10 points above random indicates severe artifacts.

Stage 2: Contrast Set Construction

Identify whether contrast sets or adversarial evaluations exist:

• Search for "[benchmark] contrast sets", "[benchmark] adversarial examples"
• Check if CheckList-style behavioral tests have been applied
• Look for counterfactual data augmentation studies

Record performance drops on contrast sets. Drops > 20 points indicate reliance on surface patterns.

Stage 3: Format Manipulation Probes

Search for evidence of format sensitivity:

• Prompt template sensitivity studies
• Label name/ordering effects
• Verbalization effects in classification
• Input length correlations with labels

Record whether minor format changes cause disproportionate score changes.

Stage 4: Conclusion Synthesis

Aggregate evidence into artifact severity assessment:

| Severity | Criteria | |----------|----------| | Critical | Partial-input baseline within 5 points of full model | | High | Contrast set drop >20 points OR format sensitivity >10 points | | Medium | Known artifacts documented but partial mitigations exist | | Low | Minor artifacts, full-input still required for high performance | | None | No evidence of artifacts (may indicate insufficient probing) |

Output

artifact_report:
  benchmark: string
  overall_severity: critical|high|medium|low|none
  partial_input_baselines:
    - input_type: string  # e.g., "hypothesis only"
      performance: float
      full_model_performance: float
      gap: float
      source: string
  contrast_set_results:
    - contrast_set: string
      original_performance: float
      contrast_performance: float
      drop: float
      source: string
  format_sensitivity:
    - manipulation: string
      score_range: string
      source: string
  shortcuts_identified:
    - shortcut: string
      mechanism: string
      exploitability: high|medium|low
  evidence_completeness: thorough|partial|minimal

Yield Report

| Metric | Minimum | |--------|---------| | Literature sources checked | 5 | | Artifact categories probed | 3 | | Evidence items collected | 4 | | Severity classification produced | 1 |