plumb-line

Every value in a program came from somewhere: a database, an API call, a test fixture, a default, a guess. Once it is sitting in a variable they all look the same. That is how a stubbed service answers "success" and the tests go green, how a guessed field flows into a report, and how a fallback an agent wrote to get a test passing ends up behind the dashboard, weeks after anyone remembers it is a fallback.

plumb-line finds those places, and can stop new ones from appearing. A set of review-time tools (a Claude Code audit skill, lint rules, git hooks and a GitHub Action) checks a repository or a pull request for mocks treated as real, guesses treated as facts, and claims with nothing behind them. A small zero-dependency library for JavaScript and Python labels each value with where it came from and how much to trust it, and keeps the label attached through every derivation: a result built from a mock says so, and no later step can upgrade it.

You probably want this if AI agents write or modify your code; if mocks, fixtures, fallbacks or synthetic values sit anywhere between an input and an output; or if your outputs are claims: a figure in a paper, a risk score, a forecast, a "safe to proceed". Common in agent-built systems, research code, data and ML pipelines, and inherited codebases. If your app reads a trusted database and shows what it finds, you probably don't need the run-time layer; the fit map says so plainly.

Install

As a Claude Code plugin. The repository is its own marketplace:

/plugin marketplace add slopstopper/plumb-line
/plugin install plumb-line@plumb-line

Then run plumb-line-adopt. It looks at your repository and tells you which parts of plumb-line fit and what to run first. Updates arrive through /plugin.

The library, with or without the plugin:

npm install plumb-line-provenance      # JavaScript
pip install plumb-line-provenance      # Python

Zero dependencies. You can also copy primitives/js/ or primitives/python/ straight into your project.

In CI. Add the GitHub Action. On every pull request it runs the checks your .plumb-line/enforcement.json manifest names and writes one SARIF log, with no agent involved. Write that manifest by hand — the shape is in ACTION.md; having plumb-line-bootstrap write it is planned, not yet proven. Uploading the log to GitHub's code-scanning tab is wired (a sha-pinned upload-sarif step), but this repo's own CI uploads nothing, so ingestion is unobserved until an adopter reports it.

Not using Claude? portable/README.md is the entry point without the plugin.

The library

One rule sits underneath all of it: combining values can keep or lower their trust level, never raise it (the combination law).

const base  = mark(1000, { source: "real", confidence: "high" });
const rate  = mark(1.25, { source: "mock", confidence: "low" });
const total = derive([base, rate], (a, r) => a * r);

total.derivedFromMock; // true   inherited from rate, and impossible to clear
total.confidence;      // 'low'  only as certain as the weakest input

mark puts the labels on a value (real, mock, inferred, fallback; confidence high down to none). derive runs your own function on labelled values and carries the labels through, keeping the weakest. The library never does the arithmetic and never changes a value; it only keeps the labels honest.

A real incident, reconstructed

A documented failure, boiled down to a few lines (the demo): a server checks five tools and reports on their health, and three of the five are stubs that always answer "success". The first run is the program as written. The second is the same program with plumb-line's labels on its values.

$ node broken/toolserver.mjs
  hash_text          success
  spawn_worker       success
  ...
system health: operational (5/5 tools succeeded)

$ node instrumented/toolserver.mjs
  hash_text          success  [source: real]
  spawn_worker       success  [source: mock]
  ...
system health: operational (5/5 tools succeeded)

report provenance:
  derivedFromMock: true
  confidence: none
  weakestSource: mock
  mock inputs: 3/5 (computed from lineage, not estimated)

attempted launder (derive with source: "real"):
  laundering: clean source 'real' but derivedFromMock is true
  source over-claim: source 'real' is cleaner than its ancestry's weakest source 'mock'

Same code, same "operational". The second version knows that three of its five results came from stubs, and when the code tries to relabel the report as real, the library refuses. That is the whole idea: a mocked result cannot be laundered into a real one.

It is the first of three documented incidents reconstructed in this repository, from three different fields:

Different fields, same pattern: information lost its status somewhere in the system, and a downstream claim was treated as stronger than its evidence. None of the reconstructions claims plumb-line would have prevented the incident; each shows where the lost status would have been visible.

How it fits together

flowchart TB
  subgraph run["Run time · deterministic"]
    direction LR
    I[your inputs] --> M["mark()"] --> D["derive()"] --> O["outputs carrying<br/>source · confidence · taint"]
  end
  subgraph review["Review time"]
    direction LR
    B[plumb-line-bootstrap] --> L["lint rules + git hooks<br/>(deterministic)"] --> G["GitHub Action → SARIF<br/>(deterministic)"]
    A["plumb-line-audit<br/>(LLM-assisted)"] --> R["report → plumb-line-remediate<br/>(opt-in)"]
  end
  run ~~~ review

Run time is the library: labels travel with values inside your own code. Review time looks at a repository or a pull request for places where a mock was treated as real, a guess as a fact, or a claim has nothing behind it. The lint rules, hooks and the Action are deterministic; plumb-line-audit is LLM-assisted and writes a findings report, which plumb-line-remediate applies only if you ask. adopt and method route you in and teach the ideas. Use either layer on its own, or both.

What the audit writes

A findings report, each finding tied to a file, a principle and a suggested fix, with a status (violation, needs-review or advisory), plus an omission-pass table that walks every output for what is missing. This is plumb-line-audit run over the broken half of the toolserver demo above with the answer key withheld from it; two of its nine findings, the suggested-fix column dropped for width:

The full report is committed as written, from its header to its format-validation line, at the v0.12.0 release harness (#576). Set against the demo's answer key, one P4 — Quarantined fakery violation covers both of the key's P4 rows, and the P6 — Maturity vocabulary row is a confirmed violation. The P3 — Confidence + provenance row is met in part: one violation says the health summary launders the stubs into a clean verdict, and the omission table marks provenance and confidence missing on the stubs, but it judges confidence not needed on the two deterministic real tools, where the key counts it missing on every result. The earlier report-format: v3 report is kept as history. The audit is measured rather than trusted; the next section says how.

What is deterministic, and what is not

The library, the lint rules, the hooks and the Action are deterministic: the same inputs always give the same result. A conformance suite holds the JavaScript and Python versions to identical behaviour, and the validation results show every planted violation caught with no false positives.

The audit and remediate skills use an LLM, so plumb-line measures them instead of trusting them. Before any release that changes them, independent auditors run them blind against test repositories with violations planted and the answers removed; a missed violation blocks the release unless a maintainer waives it in writing (the harness). For v0.12.0, every blind auditor found every planted violation (9 of 9), including in a new test repository whose tests were changed so an unmet requirement reads as met. The remediate skill failed its validation twice, both times by editing before showing its plan, and passed 2 of 2 once the plan became a file written before the first edit. 17 of 18 reports passed the format checker; the one that failed had said it was not checked (the record).

Measured on agents