/sota-testing

name: sota-testing
description: >-
  State-of-the-art software testing strategy and practice (2026) for designing
  test strategy, writing unit/integration/e2e tests, or auditing test suites.
  Covers suite shape (pyramid/trophy/honeycomb), test design quality
  (behavior-first, AAA, determinism, smells), test doubles
  (mocks/fakes/stubs), test data (builders over fixtures), real-dependency
  integration (Testcontainers-style), contract testing (Pact/consumer-driven),
  e2e/UI strategy (selectors, auto-waiting, flake economics), property-based
  testing, fuzzing, mutation testing, approval testing, and suite health/CI
  (flaky-test policy, coverage philosophy, sharding). Trigger keywords -
  testing, test strategy, unit test, integration test, e2e, end-to-end,
  coverage, flaky tests, TDD, contract testing, property-based, mocking,
  fixtures, snapshot test, mutation testing, fuzzing, BDD, Gherkin,
  given-when-then, acceptance criteria, security testing, WSTG, IDOR test,
  authz test, abuse case, DAST. Use for BOTH building and auditing test
  suites.

SOTA Testing (2026)

Expert-level, language-agnostic rules for producing and auditing production test suites. Per-language runner/tooling details (pytest, go test, cargo test, vitest/jest) live in the language skills (`sota-python`, `sota-golang`, `sota-rust`, `sota-javascript-typescript`) — this skill defines the strategy, design discipline, and quality bar those tools execute against. Every rule states the *why*; every rules file ends with an audit checklist of yes/no questions and grep-able smells.

Purpose

Two consumers, one source of truth:

• **BUILD mode** — designing a test strategy or writing tests for new code:

follow the rules as defaults, not suggestions. Deviate only with an explicit comment justifying the deviation.

• **AUDIT mode** — reviewing an existing suite: hunt violations using the audit

checklists, classify by severity, report in the finding format below.

BUILD mode

1. Before writing tests, read the rules files relevant to the layer you are testing (see index). A service touching HTTP + DB + a message queue needs `01`, `02`, `03`, `04`. 2. Apply the **top-10 non-negotiables** (below) unconditionally. 3. Decide the suite shape FIRST (`rules/01`): what counts as a unit here, where the integration boundary is, which 3–10 flows deserve e2e. Write that decision down (CONTRIBUTING.md or a test README) so the next contributor doesn't relitigate it. 4. New test code is production code: same review bar, same lint rules, no `TODO: assert something` placeholders. A merged test with no assertion is worse than no test — it manufactures false confidence. 5. Write tests alongside the code, not after the PR is "done". For bug fixes, write the failing test first — it is the only proof the fix fixes anything. 6. Default to real dependencies in containers over mocks for anything with I/O semantics you don't own (DBs, brokers, caches) — see `rules/04`. 7. When generating code for a test that legitimately violates a rule (e.g. a sleep in a test that verifies a timeout), comment why inline.

AUDIT mode

Audit the suite, not just the tests: shape, doubles discipline, data management, CI health, and what is *missing* (untested risk) all count.

**Severity conventions:**

• **Critical** — the suite lies: assertion-free tests, tests that can't fail

(always-green), mocks asserting mock behavior, disabled/skipped tests hiding known-broken production behavior, coverage gates gamed by meaningless tests.

• **High** — the suite is unreliable or unmaintainable at current trajectory:

shared mutable state between tests, order-dependent tests, real time/network/randomness without injection, flaky tests un-quarantined and retried-to-green, e2e suite owning logic the unit layer should own, mocking internals so refactors break hundreds of tests.

• **Medium** — quality erosion: mystery-guest fixtures, multi-behavior tests,

snapshot dumps nobody reviews, sleeps instead of waits, fixture data with irrelevant noise, missing negative-path tests on critical flows.

• **Low** — style/hygiene: weak names, redundant assertions, minor AAA

violations, missing parameterization of near-duplicate tests.

**Finding format** (one per line):

file:line | rule-id | severity | finding and concrete fix

Example:

tests/orders_test.py:88 | 02-determinism | High | uses datetime.now(); inject a fixed clock so the test cannot fail at month boundaries
tests/api/user.spec.ts:12 | 03-mock-boundary | High | mocks internal UserValidator; test the real validator, mock only the HTTP gateway

End every audit with: findings table, top-3 risks, and a prioritized fix list (quick wins vs structural).

Rules index

| File | Read this when... | |------|-------------------| | `rules/01-strategy-and-shape.md` | choosing pyramid/trophy/honeycomb, defining unit vs integration boundaries, deciding what NOT to test, risk-based prioritization, budgeting test cost | | `rules/02-test-design-quality.md` | writing or reviewing any test: behavior-over-implementation, AAA, naming, one logical assertion, determinism (clock/random/network — incl. **proving hermeticity by running the suite with egress blocked**, and tests that pass because a real call succeeded), test smells catalog (assertion-free, tautological, the liar, mystery guest, resource optimism), snapshot discipline | | `rules/03-doubles-and-test-data.md` | deciding mock vs fake vs stub, fixing over-mocked suites, building test data (builders/factories vs fixtures), seeding test DBs, using production data | | `rules/04-integration-contract-system.md` | testing against real DBs/brokers (Testcontainers-style), contract testing between services (Pact, schema-based), API testing, migrations, message/queue tests, ephemeral environments, and **a bench that fails silently — including a green run that quietly ran fewer tests** | | `rules/05-e2e-and-ui.md` | buil