llm-models

LLM model playbook

Model-selection and model-behavior facts for the in-app assistant. Read before changing prompts, providers, tool schemas, or model defaults. Measured claims come from `app/scripts/prompt-eval.mts` runs (temp 0, 2026-07) and carry issue or commit references; re-run the eval before and after any prompt or provider change and put both scores in the PR.

Backends

• Ollama (user's own server) is the most accurate everywhere and the head of

every platform's ranking; `qwen3:8b` is the recommended model. Below it: iOS runs llama.cpp (Qwen3 4B Instruct) on Metal then Apple Foundation Models; Android runs Gemini Nano; web runs WebLLM on WebGPU. WebLLM is gated off on iOS (WKWebView's ~2GB jetsam limit kills model load).

• llama.cpp is iOS-only. It was removed from the Android build (issue #270):

no GPU path there meant ~6.6 tok/s decode against Gemini Nano's ~1.5s replies, for 76MB of native libraries and a 2.5GB model download. Do not reintroduce it without a working Vulkan/NPU path and a measured win.

• Gemini Nano (Android system model, ML Kit GenAI Prompt API over AICore) is

the Android system backend. Unlike Apple's it cannot constrain output: ML Kit structured output is compile-time KSP codegen with no union type, so no per-tool turn schema and no removable answer branch. It is driven with the llama.cpp text contract instead. No native tool calling either.

• The two system models, measured 2026-07-28 on the same 90 cases through each

backend's real production path: Gemini Nano (Pixel 10) plans 33/38 and reads time 48/52; Apple Foundation Models (iPhone 17 Pro Max, iOS 26.5.2) plans 21/38 and reads time 47-51/52. Nano is far better at planning; they are level on time. The no-tool half is the sharpest split: Nano 8/8, Apple 4/8, with Apple calling `list_events` three times for "who won the world cup in 2018" and `get_event` for "delete event 1234".

• Apple's planning deficit is NOT a prompt problem, and the trimmed prompt is

not what causes it. Giving it the whole shared system prompt plus whole tool descriptions on the planning turn moved 21/38 to 23/38, inside its own run-to-run spread, and introduced failures that were not there before: arguments leaking between cases (a German window on two English questions) and junk in `objectType` ("NO", a tag name). Both attempts were reverted. What the extra text did fix is narrow and real: `count_events` chosen for a calendar window went from four failures to one, because that distinction lives past the first sentence of those two descriptions. Whether the rest is the model or the 4096-token window is unseparated.

• Apple's output SHAPE is flawless and its judgement is not: every one of its

planning failures was a well-formed, schema-valid tool call that was simply the wrong call. Constrained decoding fixes shape and buys nothing else.

• AICore rate-limits a burst (`ErrorCode.BUSY`), which is NOT the plugin's own

concurrency guard and must not share its code: running the contract eval straight after the time eval got every case rejected and reported 0/14 as if the model had failed them all. `RATE_LIMITED` is its own code now and the contract eval backs off and retries; a run reports how many retries it took (11 on the first clean run).

Measured model floor (prompt-eval, temp 0, 2026-07)

• `qwen3:8b`, `qwen3:30b-a3b`, `qwen3:4b`: 33/33 native tool-calling,

24/24 constrained envelope. `qwen3:8b` is the recommended Ollama model.

• `llama3.2`: 33/33 native, 21/24 envelope.
• `qwen3:1.7b`: 21/33; below the 4b tier models fail the harness and are

not worth supporting.

Reasoning control

• The Ollama tag `qwen3:4b` resolves to Thinking-2507, whose reasoning

cannot be disabled; use `qwen3:4b-instruct` when reasoning is unwanted.

• `/no_think` in a prompt is a placebo on Ollama: it hides the tag and

still reasons at full latency. The real switch is `reasoning_effort: "none"` on the /v1 endpoint (about 5x faster, same eval scores); the adapter sends it only to confirmed-Ollama servers.

• WebLLM disables thinking with `extra_body: { enable_thinking: false }`.

Patterns by model size

• Small and on-device models ignore prompt-only guardrails. Structural

fixes are the ones that hold: the turn schema makes the answer branch unreachable without a real tool result (4ee2bbbf), failure paths append a correct-and-retry guard instead of echoing raw errors (0a720c01).

• Small models copy phrases perfectly but fail direct field-filling: time

windows used copy-interpret-compute (refs #265; direct fills then scored 27/36 qwen, 15/36 llama). SUPERSEDED for time (refs #444): copying itself proved the fragile step ("same day, last week" copied as "last week"), and the rebuilt interrogation - whole question in, a windows ARRAY of meaning-first branch shapes out, rolling branch offered only when the question shows a rolling marker, code resolving every window - scores 124/124 across every time class on qwen3:8b, comparisons included. The copy pattern still holds where it never broke (place words). On-device backends run the same path but are UNMEASURED there (array-of-anyOf schemas untested on Apple FM / Gemini Nano): run the settings time eval on-device before trusting those numbers.

• Prompt classification teaches dimensions (intent by subject), never

instance lists; instance-based triage misclassified every combination outside its examples, four times.

• A name enum cannot express abstention: asked which monitor a place means,

qwen3:8b substituted the nearest listed name for a place the list lacks (12/16) under three prompt wordings and both enum orders. What worked (refs #427, temp 0, 2026-09): copy the place words first, then answer `covered` as its own boolean, then the names, with the no-coverage verdict derived in code from place+covered (21/24; time words the model copies into `place` are stripped in code via `s