agentic-chatops

agentic-chatops

AI agents that triage infrastructure alerts, investigate root causes, and propose fixes — while a solo operator sleeps.

For the complete technical reference, see README.extensive.md.

The Problem

One person. 310+ infrastructure objects across 6 sites. 3 firewalls, 12 Kubernetes nodes, self-hosted everything. When an alert fires at 3am, there's no team to call. There never is.

The Solution

Three agentic subsystems that handle the detective work — ChatOps (infrastructure), ChatSecOps (security), ChatDevOps (CI/CD) — built on n8n orchestration, Matrix as the human interface, and a tiered agent architecture (deterministic triage scripts → Claude Code → human). The human stays in the loop for every infrastructure change: the system never acts without a thumbs-up or poll vote, and since 2026-06-09 a remediation proposal cannot even reach the approval poll without a machine-computed consequence prediction attached (see Infragraph below).

What Makes This Different

Self-Improving Prompts — now with A/B trials (nobody else does this)

The system evaluates its own performance and auto-patches its prompts. Every session is scored by an LLM-as-a-Judge on 5 quality dimensions (gemma3:12b local-first since 2026-04-19; max-effort calibration via gw-mistral-large on the shared LiteLLM). When a dimension averages below threshold over 30 days, the preference-iterating patcher (IFRNLLEI01PRD-645, 2026-04-20) generates 3 candidate instruction variants (concise / detailed / examples) and assigns each future matching session to one arm via deterministic BLAKE2b hash — plus a no-patch control. A daily cron runs a one-sided Welch t-test once every arm reaches 15 samples; the winner is promoted only if it beats control by ≥ 0.05 points with p < 0.1. Otherwise the trial is aborted. Prompt-level policy iteration — no model weights are ever fine-tuned.

Session → LLM Judge (5 dims) → dimension trending below threshold
  → prompt-patch-trial.py generates 3 candidate variants + 1 control
  → future sessions hash-routed to arms → Welch t-test at 15+ samples/arm
  → winner promoted to config/prompt-patches.json (source: "trial:N:idx=I")
  → next eval cycle scores the new patch → loop continues

Infragraph — a Causal World Model with a Non-Bypassable Prediction Gate (2026-06-09)

The system maintains a causal dependency graph of the entire infrastructure (361 nodes / 468 edges in the causal layer; 721 entities / 661 relationships in the combined GraphRAG+infragraph knowledge graph) seeded daily from five truth layers — live Proxmox cluster API (0.95 confidence), LibreNMS dependency parents (0.90), NetBox devices + physical cables (0.85–0.90), operator-declared edges, and a statistical incident-co-occurrence miner deliberately capped at 0.75 — with per-edge dynamics (expected alert cascades, propagation delays, recovery times) learned from 159 chaos experiments and the full triage history. This is a genuine model-free → model-based shift enforced in control flow, not data:

1. Prediction is computed outside the LLM — deterministic graph traversal (infragraph-query.py), called by the n8n orchestrator, never at the model's discretion.
2. Prediction is mandatory — the Runner commits a plan-hash-keyed prediction artifact before any approval poll; a remediation proposal without one is rewritten to [POLL-WITHHELD:NO-PREDICTION] and demoted to analysis-only. The kill-switch (INFRAGRAPH_DISABLED=1) fails the remediation lane closed.
3. Verification is mechanical — after execution, code (never the LLM that proposed the action) diffs observed alerts against the prediction and writes a match / partial / deviation verdict; deviation = surprise = never auto-resolve.

The eval is falsifiable by design: a degree-preserving shuffled-graph negative control runs alongside every prediction. The 2026-05-11 cascade backtest passed the criterion (control ratio 0.367 ≤ 0.5×) only after four honest iteration rounds, each driven by what the misses revealed. Suppression authority is granted per rule by the operator — the system proposes (control YouTrack issue with evidence table), the human approves, and closing the control issue instantly revokes. Runbook: docs/runbooks/infragraph.md.

Autonomy-Forward Gate — Human as Circuit-Breaker, not Gatekeeper (2026-06-16)

Most "human-in-the-loop" systems assume the human is watching. Ours measured that the operator had voted on almost none of the approval polls in the prior two months — so the loop was a dead-end: reversible work stalled on a 30-min pause and genuinely-critical work paged no one. The fix is a 3-band risk gate (classify-session-risk.py): reversible, Infragraph-prediction-backed changes auto-resolve; a tightly-scoped critical set (HIGH-risk, P0-host blast, irreversible, model deviation) is the only thing that pages the operator by SMS. The safety floor is mechanical and non-configurable, and the whole gate flips on/off with a single touch/rm of a sentinel file — no workflow edit, instant kill-switch. Runbook: docs/runbooks/risk-based-auto-approval.md.

Self-Verifying Reliability Layer — the system watches itself (2026-06-21)

The defining failure mode here was never a crash — it was months of silent darkness (the auto-resolve pipeline dead across 5 layers, scanners dark 5 weeks, an apiserver crash-looping 27 days) where nothing alerted because standard alerting treats no data as no problem. So the autonomy loop is now a continuously-verified subsystem:

• Control-plane dead-man's-switch — gateway-watchdog.sh emits a heartbeat every 5 min via a trap … EXIT; a Prometheus alert with an absent() clause pages by SMS if the heartbeat goes stale or vanishes (node_exporter/host down). It watches the thing that watches the pipeline.
• Synthetic-incident canary — a daily probe drives the real classify→predict spine end-to-end against an isolated throwaway DB, so it proves the spine is alive (3 stages + plan-hash coherence) while structurally being unable to pollute production, collide a real fail-closed gate, or trigger remediation. A tier-1 SMS fires if it ever leaks a row into the live DB.
• False-auto-resolve governance — the system measures its own root-cause discipline: a pattern it auto-resolved that recurs within 24h is a false-auto-resolve, and a repeat offender (≥3×/30d) is auto-demoted so the gate escalates it instead of auto-closing it again — automatically, reversibly (30-day expiry), with no human review (human-as-circuit-breaker, not gatekeeper). Intentionally-suppressed flappy alerts are excluded so it never re-introduces suppressed noise.
• Bi-temporal knowledge — infragraph edges and compiled-wiki facts carry a contradiction/supersession axis with time-since-confirmation decay (reporting-only — it flags edges for re-ratification, never silently changes a prediction).
• Self-learning scheduled-reboot suppression (2026-06-29) — hosts with a discovered and promoted reboot schedule (observe-≥2-boots-before-live, strict DST-correct cron windows) get their on-schedule reboot alerts suppressed before any session spawns, with a two-phase verify that reopens + pages if the boot wasn't a clean systemd-reboot. Safety floor: critical-never, allowlisted rules only, sentinel kill-switch, fail-open. Runbook: docs/runbooks/scheduled-reboot-suppression.md.

Orchestrator Control-Plane — the system governs itself (2026-06-26)

The agentic federation grew to ~10 subsystems and 363 components (320 at the 2026-06-26 landing) — Cronicle jobs, 57 n8n workflows, hooks, and the RAG / infragraph / teacher / chaos subsystems — coordinated only by convention, a shared SQLite, and the Prometheus textfile bus. Nothing owned their liveness as a set, and a 2026-06-25 audit proved the cost: MemPalace hooks, the OTel span sink, the tool-call log, and even the self-audit itself had run dark for weeks-to-months, each invisible because standard alerting reads no data as no problem. The fix is a thin governing layer (IFRNLLEI01PRD-1421) — three bricks built on the existing Prometheus + SQLite substrate, no platform rewrite (the research explicitly rejected adopting LangGraph / Temporal / Airflow / Dagster / Backstage):

• Component Registry (scripts/registry-check.py) auto-discovers all 363 components (199 cronicle-job + 77 prom-writer + 57 n8n-workflow + 28 db-table + 2 cron, as of 2026-07-08), each with a declared liveness expectation — 15 critical, 0 critical-dark, ~10 known-dark-by-design. The dark-component failure class is now caught mechanically (RegistryCriticalDark, tier-1 SMS) instead of by a manual quarterly sweep.
• Interaction Graph (scripts/interaction-graph.py) static-analyzes 313 scripts into a read/write asset graph (Dagster's model in ~250 lines): currently 0 GAPs (the Session-End → reconcile orphan-consumer hole that silently darkened 4 analytics tables is closed), 0 cron-clashes, and 23 multi-writer conflicts surfaced for review.
• Orchestration Benchmark (scripts/orchestration-benchmark.py) replays a synthetic incident stream through the isolated classify→predict spine and scores 4 orchestration invariants — score 1.0, 4/4, including safety-composition: an irreversible incident is never auto-resolved, verified across the whole stream rather than case-by-case (OrchestrationSafetyFailure, tier-1).

All five rules are live in-cluster (infra MRs !347 + !348), and a fault-injection drill proved the alerts actually fire, not merely evaluate. The control-plane monitors its own three bricks — the who-watches-the-watcher gap is fully closed.

Plane-A self-healing platform controller — the actuator half (2026-06-26). The bricks detect; a Kubernetes-style self-healing operator (scripts/platform-controller.py, */5, armed) acts — closing the loop the absent human left open. It heals only idempotent platform operations: reactivate an inactive critical n8n workflow (it monitors all 57), re-run a failed safe-list Cronicle job, restart Cronicle, plus a consolidated watchdog heal-library. Heals are rate-limited by exponential heal-backoff → CrashLoopBackOff → SMS escalation, exactly as a k8s controller would. Crucially it draws the same Plane-A / Plane-B line k8s does between keeping pods alive and deciding app logic: Plane-A keeps the platform alive (crons, Cronicle, bricks, writers, n8n); Plane-B is the mission (resize a VM, reboot a host, resolve an incident) — the controller NEVER touches B. That stays the autonomy-forward lane's job. It consolidated the standalone watchdog into one operator, and carries its own dead-man.

Cronicle scheduler — every job has run-history now (2026-06-26). All cron jobs (180 at migration: 107 gateway + 72 agora-quant; 199 registered as of 2026-07-08) migrated off raw crontab to a native Cronicle scheduler: per-job run history, per-job-death alerting (the gap a flat crontab can't see — a crontab line that silently stopped firing looks identical to one that never existed), a REST API the registry seeds from, and auto-quarantine of a repeatedly-failing job.