pickysteve

▶ Watch the trailer

https://github.com/user-attachments/assets/8750946b-36be-4c48-bf73-79513451d1f5

PickySteve is a lightweight orchestration layer. A cheap model figures out which skill a request actually needs, retrieves that one skill, and hands a small, focused, untrusted-data-boundaried context bundle to a capable model. It does not dump every tool and document you own into context on every request.

This repo is Phase 1 (MVP), built to an architecture spec. Phase 2 work (tracing platform, standing eval harness, credential vault, sandbox) is not built yet. Each piece gets added only when a real Phase 1 failure justifies it.

30-second quickstart

# from the repo root (uv 0.10+; on Windows the venv python is .venv/Scripts/python.exe — substitute it throughout)
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -r requirements.txt

# choose your model — local Ollama, OpenAI, Claude, OpenRouter, or any OpenAI-compatible endpoint
.venv/bin/python -m pickysteve.setup

# calibrate the reranker floor on the labeled set
.venv/bin/python eval/calibrate.py

# run one request
.venv/bin/python -m pickysteve "review my Rust endpoint for security and REST design"

Bring your own model. python -m pickysteve.setup asks which model to use and saves it. Runs on anything that speaks the OpenAI-compatible API: local Ollama (offline, no key), OpenAI, Claude, Gemini, Llama, etc. via OpenRouter / LiteLLM / their native compat endpoints. The published benchmarks were measured on local qwen3:8b; a different model just needs a eval/calibrate.py re-run.

Note: this is currently a uv / git clone install. There is no PyPI package yet, so uvx pickysteve and pipx install pickysteve do not exist. If that changes, this section gets a one-liner. For now, the fastest path to a real coding agent is the connector installer below.

Wire it into your agent (one command)

python -m pickysteve.connectors.install --list   # see which of 18 agents are detected
python -m pickysteve.connectors.install --all    # wire every detected agent (backs up configs first)

Supports Claude Code, Codex, Cursor, Windsurf, Cline, Roo Code, Gemini CLI, Qwen Code, Goose, OpenHands, GitHub Copilot, Kimi Code, OpenCode, ZeroClaw via MCP stdio, and Aider, Hermes, OpenClaw, NanoClaw via an OpenAI-compatible proxy on :8077/v1. Full per-agent config snippets and the connectivity matrix are in INTEGRATIONS.md.

How it works

flowchart TD
    A[Request] --> B[Security Gate\nscan raw request]
    B -->|clean| C[Router\ncheap model → search query]
    B -->|injection| X1[Abort]
    C --> D[Retrieval\nBM25 + embeddings, RRF fused]
    D --> E[Security Gate\nscan every retrieved doc]
    E -->|clean| F[Rerank\ncross-encoder vs original request]
    E -->|poisoned| X2[Abort / drop candidate]
    F --> G[Floor + Dedupe\nbelow floor → clarify, don't guess]
    G --> H[Knowledge Graph\nconfused_with edges + distinguishers]
    H --> I[Judge\nLLM reads full skill bodies + KG notes]
    I --> J[Compat Check\nflag conflicts, don't merge]
    J --> K[Assembly\nnonce-wrapped untrusted-data boundary]
    K --> L[Execution\ncapable model does the work]
    L --> M[Log\nfull trace to logs/runs.jsonl]

Ten stages: gate, route, retrieve, gate again on retrieved content, rerank, floor/dedupe, knowledge-graph context, judge, compat-check, assemble, execute, log. The second gate pass scans every retrieved candidate, not just the user's request. Most similar projects skip that pass, and it is the highest-risk surface: a poisoned skill doc is attacker-controlled content sitting right next to your execution model.

The stack (and why)

Role	Choice	Note
Runtime	Python 3.11 via uv	The default Python here is 3.14, which still has shaky torch wheels. uv pins an isolated 3.11 venv where the ML stack is stable.
Security gate	stackone-defender[onnx]	The real StackOne defender (Python port, v0.7.2), not a regex placeholder. Bundled ~22MB ONNX classifier, no download.
Router / compat / clarify / execution	local Ollama qwen3:8b via the native /api/chat (think:false)	Runs with no cloud key. The OpenAI-compat endpoint does not honor thinking control for qwen3 (it dumps output into a reasoning channel and leaves content empty, roughly 20x slower), so the client uses the native endpoint by default. Set PS_OLLAMA_NATIVE=0 / PS_LLM_BASE_URL for any OpenAI-compatible host.
Retrieval	rank_bm25 + sentence-transformers embeddings, fused with RRF	Hybrid keyword + dense.
Reranker	BAAI/bge-reranker-base cross-encoder	Exactly the model the spec names. Its output is a logit, not a probability, so the floor is calibrated rather than guessed.
Logging	flat JSONL	Manual review is the Phase-1 eval process.

Total Phase-1 dependencies: stackone-defender, rank-bm25, sentence-transformers, openai, numpy. That is the minimal set the spec prescribes.

Two decisions the spec left open (decided and documented)

• Retrieval unit (§2.3): each markdown file is one retrieval unit. A skill folder with several files (see registry/rag-architecture/) yields multiple units sharing a skill_id. After reranking, units from the same skill collapse to the best one in assembly, so the execution model never receives three chunks of one skill.
• Gate policy on a poisoned retrieval (§2.1): default RETRIEVED_INJECTION_POLICY=abort. If a retrieved candidate trips the gate (high-risk), the whole request aborts. The documented alternative is drop, which discards just that candidate and continues. For allowed-but- sanitized content, the pipeline uses the Tier-1-sanitized text downstream (defense in depth) and logs that sanitization happened.

Refinements after a 21-agent adversarial review

The first validation surfaced three failures. Fixing them, and adversarially reviewing the fixes, added these mechanisms. See FINDINGS.md for the full before/after.

• Tier-3 escalation (gate, request path only): a legitimate question about prompt injection was being blocked. The request gate now enables the defender's Tier-3 LLM hook over the gray band [0.64, 0.85), just above the model's calibrated 0.64 block threshold. A cheap adjudicator can rescue a would-be block but never flip a would-be allow, while near-certain attacks (≥0.85) still hard-block without consulting it. Retrieved third-party content never escalates (strict gate).
• Multi-intent router with §2.4-safe rescue: the router emits sub-queries and retrieval unions across them for recall. Reranking stays governed by the original request (§2.4). Only a genuinely compound request (two or more distinct sub-intents) also maxes over its sub-queries, to surface a secondary intent the full-request score would bury.
• Relative-dominance gate: a secondary skill is kept only if it scores at least DOMINANCE_RATIO (0.08) times the top skill. This keeps PickySteve picky instead of dumping marginal tag-alongs.
• Honest #13 fix: a correct skill that the reranker under-scored was fixed by enriching the skill doc with real symptom vocabulary, not by lowering the floor onto leaked data. The floor is recalibrated on a leakage-free labeled set with hard-negatives.

Benchmarks

All numbers below come from this repo's own eval docs and logs.

Routing accuracy, the trifecta (DEEP_CONTEXT.md):

Suite	Tasks	Result
Base	26	100% × 10 consecutive runs (qwen3 judge)
Harder (base + 16 brutal adversarial)	42	100% × 10 (qwen3 judge)
Held-out (unseen, fresh confusion mechanisms)	47	100% × 10 (Claude blind judge)
Heldout2 (hardest, deliberately unsaturated adversarial set)	24	23/24 (96%). One genuine miss on a compound canary/feature-flag task where the trap ranked above gold (logs/heldout2_final_run.log)

The heldout2 set is kept deliberately hard and unsaturated. New confusable-pair tasks get added faster than the router/rerank stack is re-tuned, so it acts as a running canary for regressions rather than a suite that is expected to hit 100%.

On a 40-request held-out accuracy set with no calibration overlap (TEST_REPORT.md): 90% overall correct, 100% top-1 accuracy (30/30 answerable), 96.7% full recall, MRR 1.000, 100% off-domain rejection (haiku/recipe requests correctly get no_confident_match).

Two-tier gate (recall-all + conformal abstention). The cheap local judge routes singleton predictions directly; ambiguous cases escalate to a frontier judge (logs/two_tier.out):

Metric	Result
Conformal coverage	44/47 = 94%
Routed cheap (singleton)	29/47 = 62%, correct 27/29
Escalated to frontier	18/47 = 38%, correct 18/18
Combined top-1	45/47 = 96%

Security, red-team detection (SECURITY_AUDIT.md, TEST_REPORT.md):

• 180-payload corpus (129 attack / 51 benign, 14 evasion families): 100% attack detection, zero bypasses after hardening. The baseline was 86%.
• Separate 115-attack corpus: 97.6% detection on the request path, 96.5% on retrieved content, up from 87.1%. Benign false-positive rate held at 0.0% throughout.
• On the adversarial 180-payload corpus, benign-allow rate is 61% (39% false-positive on deliberately tricky security-flavored prompts). On the real skill registry, false positives are 0/43, verified by a startup-time warm pass the server refuses to serve without.

Trap-registry ranking test (SIM_REPORT.md): 24 skills built to confuse a naive matcher, 14 tasks. The gold skill outranked every trap 13/13 (100%), correct top-1 on 12/13 answerable tasks, correct no-match handling 1/1.

Why not just RAG or LangGraph?

• It does not retrieve everything and let the model sort it out. The floor, dedupe, and dominance-ratio gate exist so the execution model never sees marginal tag-along documents. The goal is picking one thing, not five plausible things.
• It is not a bigger orchestration framework. There is no state machine and no LangGraph-style graph runtime. Phase 1 is five Python modules (retrieval.py, rerank.py, router.py, security_gate.py, pipeline.py). See "Phase 1 non-goals" below for what is left out (no knowledge-graph-as-default, no standing eval harness, no sandbox) until a real failure justifies adding it.