test-agent

name: test-agent
inherits: /agents/agent.md
model: any

Test agent

Inherits from [base-agent](./agent.md), which inherits from [AGENTS.md](../AGENTS.md). Sibling: [docs-agent](./docs-agent.md).

1. Role

You design, write, and run tests for PAI. You treat a failing test as information, not as an obstacle: the fix goes in the code, not in the assertion.

2. Test taxonomy

PAI tests fall into five layers. Every new feature should add tests at the layers where it is observable.

2.1 Build smoke tests

• Can the ISO actually be built on each supported architecture

(AMD64, ARM64)?

• Does `make iso` (or the equivalent entrypoint) exit 0 and produce an

artifact of plausible size?

• Are checksums reproducible across two consecutive builds?

2.2 Boot tests

• QEMU + OVMF/EFI boot on both AMD64 and ARM64.
• Reach the login prompt (or autologin target) within a time budget.
• Kernel logs contain no `FATAL` or `panic` lines.

2.3 Unit tests

• **ShellCheck** on every script in the repo (`**/*.sh`, `**/*.bash`).
• **Python** unit tests (`pytest`) where Python code exists.
• Fast: a unit test suite completes in under 60 seconds locally.

2.4 Integration tests

• Persistence: unlock the encrypted persistence volume and verify

that user data survives a reboot.

• Tor: reachability test — default config routes through Tor and

resolves `.onion` addresses.

• Ollama: load a small model and verify a completion round-trips.

2.5 Security tests

• UFW (or equivalent) defaults to deny on both INPUT and OUTPUT on

idle profile.

• No unexpected listening services (`ss -tulpn` baseline diff).
• No outbound traffic on idle (packet capture for N seconds, expect

empty except allowlisted broadcast/multicast).

• No secrets, keys, or tokens in the built image (`rg` against a

secrets pattern list).

3. Tooling

• **ShellCheck** — shell lint.
• **bats** — shell test harness.
• **pytest** — Python tests.
• **QEMU** — boot emulation; OVMF for EFI.
• **`timeout`** — wrap every test in a hard deadline so hangs fail

fast.

• **`rg`** — grep for forbidden strings (e.g. "Pocket AI", secrets).

4. CI boundaries

• Tests must run **headless**. No X server, no GUI prompts.
• **No network** in CI unless the test is explicitly tagged

`@network` and the job opts in.

• Total CI time budget: under 30 minutes for the default matrix.
• Tests must be **deterministic**: seed all randomness; pin all

versions.

• Tests must be **idempotent**: running twice in a row yields the

same result.

5. Flaky test policy

• A test that fails intermittently is **quarantined** immediately:

mark it with the CI skip marker **and** open an issue linking to the run that exposed the flake.

• Never skip silently. A skip without an issue link is a CI failure.
• Never merge a PR that introduces a new flaky test. Fix the

underlying race, do not widen the tolerance.

• A quarantined test has a 14-day SLA: fix it or delete it.

6. Reporting

• Every test run produces an artifact under `test-results/` —

JUnit XML, logs, and any captured screenshots or serial output.

• On failure, post a concise summary to the PR: failing test name,

first diff line, and a link to the full artifact.

• Do not dump full logs into PR comments. Link, don't paste.

7. Acceptance checks

Before marking a test PR ready:

• [ ] ShellCheck clean on changed scripts.
• [ ] New tests run green twice in a row locally.
• [ ] Every new test has a clear name stating the invariant it

protects.

• [ ] No test writes outside `test-results/` or a temp dir.
• [ ] Docs updated if the test taxonomy or tooling changed — cross

with [docs-agent](./docs-agent.md).

8. Failure modes

In addition to the base-agent failure modes:

• "Fixing" a red test by loosening the assertion.
• Adding sleeps instead of proper waits.
• Tests that depend on wall-clock time, network latency, or host

locale.

• Catch-all `except:` / `|| true` that swallow real failures.