/local-llm

name: local-llm
description: Use when working with this machine's local LLM setup (LM Studio + mlx-dspark) -- checking status, changing context window, diagnosing a reasoning hang or dead request, understanding RAM/speed tradeoffs, or wiring a new script to the local inference endpoint.
domain: ops

Local LLM Setup (LM Studio + mlx-dspark)

Announce at start

"I'm using the local-llm skill to work with the local inference setup."

Architecture

Two local LLM backends exist on this machine, serving the same underlying model family (`Qwen3.8-27B`, 4-bit, hybrid attention -- see RAM cheatsheet below):

| Backend | Port | Role | Managed by | |---|---|---|---| | **LM Studio** | 1234 | Interactive chat UI use only | LM Studio app itself (own TTL-based auto-unload) | | **mlx-dspark** | 8090 | Production -- everything `build_local.py` and any script under `systems/superintelligence/*` talks to | `launchd` (`~/Library/LaunchAgents/com.local.mlx-dspark.plist`) |

`build_local.py` (all 6 copies under `systems/superintelligence/{data-analytics, finance,gtm,risk-compliance,strategy,trading}/scripts/`) talks **only** to mlx-dspark. LM Studio is not in that pipeline's path at all -- it exists purely for manual/interactive use. Do not assume both are always loaded; see "Idle-unload" below.

Config lives outside the repo (never committed, never touched by git):

• `~/.config/mlx-dspark/start.sh` -- the server launch command (model, mode,

context window, batch size, host/port, API key).

• `~/.config/mlx-dspark/api_key` -- 0600-perms API key file. `build_local.py`

reads `MLX_DSPARK_API_KEY` env var first, falls back to this file.

• `~/.config/mlx-dspark/idle_watcher.py` -- the idle-unload poller (see below).
• `~/Library/LaunchAgents/com.local.mlx-dspark.plist` -- the server's LaunchAgent

(`RunAtLoad` + `KeepAlive(Crashed)` -- verified to auto-restart on a crash).

• `~/Library/LaunchAgents/com.local.mlx-dspark-idle-watcher.plist` -- the

watcher's own LaunchAgent (same pattern, negligible RAM, just a polling loop).

The prefill trick (why this matters)

Both LM Studio and mlx-dspark, when hit via `/v1/chat/completions`, silently ignore every attempt to suppress reasoning for this model (`reasoning.effort`, top-level `reasoning`, camelCase variants, `chat_template_kwargs`, and even a persisted per-model load config -- all tried, none worked). The model just burns its entire `max_tokens` budget "thinking" and the caller sees total silence until a timeout fires. This was the original "thinks then dies after 5 minutes" bug report.

**Fix:** use the raw `/v1/completions` endpoint (not chat/completions) and hand-build a ChatML prompt with the assistant turn's `<think>` block **pre-closed and empty**:

<|im_start|>user
...<|im_end|>
<|im_start|>assistant
<think>

</think>

Since reasoning is already "closed" in the prompt itself, the model has no room left to think and goes straight to the real answer, giving the full token budget to the answer instead of unbounded reasoning. This is exactly what `_raw_prompt()` in every `build_local.py` does -- see that function's docstring for the full investigation trail. Works identically against both backends since they share the same jinja chat-template mechanics.

If you're wiring a **new** script to either backend, reuse this pattern -- don't call `/v1/chat/completions` and hope reasoning-suppression flags work.

Idle-unload (don't keep the model hot-loaded when nothing is working)

Neither backend should sit fully loaded in RAM 24/7 if nothing is using it.

• **LM Studio**: has its own TTL (currently 60m/1h) that auto-unloads an idle

model loaded via the chat UI. No extra config needed -- just don't rely on it staying loaded indefinitely.

• **mlx-dspark**: `idle_watcher.py` polls `/metrics`'s `requests` counter every

60s; if it hasn't changed for 15 minutes (env var `DSPARK_IDLE_TIMEOUT_SEC`, default 900) **and** `/health` shows a model loaded, it calls `POST /admin/unload`. This frees the full target+drafter footprint (~20GB).

**What happens to a request that lands while unloaded:** `build_local.py`'s `llm()` catches the resulting `HTTP 503` (`"no model is loaded"`), calls `POST /admin/load` once (re-supplying only `model`+`mode` -- `context_window` is **sticky across loads** server-side, no need to resend it), and retries the request once. This is invisible to callers except for one cold-load delay. Measured empirically on this machine (weights warm in OS page cache): a reload-and-retry completed in ~7s total. A truly cold boot (fresh page cache, e.g. right after a machine restart) would be slower -- budget more like 20-40s the first time until you've measured it fresh.

If you're calling mlx-dspark from something that ISN'T `build_local.py` (a one-off script, a curl command, etc.), you don't get this retry for free -- either reuse `_dspark_request`/`_dspark_load` from `build_local.py`, or catch 503 yourself and POST `/admin/load` before retrying.

RAM cheatsheet

The loaded model (`mlx-community/Qwen3.8-27B-4bit`, HF architecture `Qwen3_5ForConditionalGeneration`) is a **hybrid attention** model: only 16 of 64 layers use full (quadratic-KV-cost) attention; the other 48 use linear attention with ~constant memory cost. That makes its KV cache scale far better with context length than a plain transformer, and is why 64k context is cheap here when it wouldn't be on an all-full-attention model of this size. `max_position_embeddings: 262144`, so 64k (and even 128k) are both well within the model's trained range, not extrapolation.

| Context window | KV cache / request | x4 concurrent (`--max-batch 4`) | |---|---|---| | 32,768 | 2.0 GB | 8 GB | | 65,536 | 4.0 GB | 16 GB |

The context window itself is a mutable runtime setting (changed via `~/.config/mlx-dspark/start.sh`, see "Changing the context window" below), so it is not restated here as a fixed row. Read the live value from the `context_window` field at `curl -s http:/