/OpenMythos-Skill

name: openmythos
description: Use this skill when the user is working with the OpenMythos codebase — a PyTorch implementation of a hypothesized Recurrent-Depth Transformer (RDT) architecture by Kye Gomez. Trigger on any of these signals: files named `main.py` in an `open_mythos/` directory; imports like `from open_mythos.main import OpenMythos, MythosConfig`; variant helpers `mythos_1b`/`mythos_3b`/`mythos_10b`/`mythos_50b`/`mythos_100b`/`mythos_500b`/`mythos_1t`; mentions of `MythosConfig`, `RecurrentBlock`, `LTIInjection`, `ACTHalting`, `MoEFFN`, `MLAttention`, `GQAttention`, `LoRAAdapter`; discussion of OpenMythos-specific concepts like the Prelude/Recurrent/Coda three-stage layout, the `h_{t+1} = A·h_t + B·e + Transformer(h_t, e)` update rule, spectral-radius-less-than-one stability via the `exp(-exp(...))` reparameterization, loop-index embeddings on a `dim // 8` channel slice, the ACT remainder trick with `still_running` gating, or the `n_shared_experts` + `n_experts_per_tok` MoE scheme the repo uses; training scripts involving FineWeb-Edu and a looped model; and debugging symptoms that are specific fingerprints of looped training (residual explosion, step-reproducible loss spikes from injection-parameter spectral drift, overthinking degradation past the convergence point). A filename or a matching symbol is sufficient — the user does not need to explicitly say "OpenMythos". Do not trigger for generic transformer, generic MoE, or generic ACT questions that don't involve OpenMythos's specific implementation.

OpenMythos

OpenMythos is an open-source PyTorch reconstruction of a hypothesized **Claude Mythos** architecture, written by **Kye Gomez** ([github.com/kyegomez/OpenMythos](https://github.com/kyegomez/OpenMythos), MIT license). It implements a **Recurrent-Depth Transformer (RDT)** with three stages — **Prelude** (standard transformer blocks, run once), a **Recurrent Block** (one TransformerBlock looped up to `max_loop_iters` times with input injection at every step), and a **Coda** (standard transformer blocks, run once). Attention is switchable between GQA and MLA; the FFN inside the recurrent block is a fine-grained MoE with always-on shared experts.

The project is an *independent, theoretical* reconstruction. It is not affiliated with Anthropic. The README is careful with language like "suspected", "likely", and "most probable class of solution", and so is this skill — don't claim this is what Anthropic actually does internally. If the user conflates OpenMythos with real Claude internals, gently correct them.

This skill turns Claude into a careful senior engineer who knows this specific repo. That's the entire job. (There is an optional experimental appendix at the bottom for users who want Claude to roleplay reasoning in the RDT's Prelude → Loop → Coda shape, but it is off by default.)

The repo at a glance

open_mythos/
├── main.py         — MythosConfig, OpenMythos, all nn.Module classes (RMSNorm, GQAttention,
│                     MLAttention, MoEFFN, Expert, TransformerBlock, LoRAAdapter, LTIInjection,
│                     ACTHalting, RecurrentBlock), RoPE helpers, loop_index_embedding
├── variants.py     — mythos_1b / 3b / 10b / 50b / 100b / 500b / 1t preset configs
├── tokenizer.py    — MythosTokenizer wrapper (defaults to openai/gpt-oss-20b via HF)
└── __init__.py     — public re-exports

training/3b_fine_web_edu.py  — reference training script (DDP-ready via torchrun, FineWeb-Edu)
tests/                        — test_main.py, test_tokenizer.py, bench_vs_transformer.py,
                                small_benchmark.py, test_rope_debug.py
docs/                         — open_mythos.md (full class reference), datasets.md
examples/                     — moda_example.py, variants_example.py
example.py                    — minimal end-to-end sanity script at repo root

The forward pass — hold this in your head

input_ids
  ↓ embed
  ↓ Prelude: prelude_layers × TransformerBlock (dense SwiGLU FFN, no MoE)
  e = x  ← encoded input is frozen here, re-injected every loop
  ↓
  RecurrentBlock (one block, looped up to n_loops times; uses MoE FFN):
    for t in range(n_loops):
        h_loop = loop_index_embedding(h, t, dim//8)   # RoPE-like signal on a slice of channels
        combined = RMSNorm(h_loop + e)                 # input injection into normed stream
        trans_out = TransformerBlock(combined) + LoRAAdapter(trans_out, t)   # per-depth LoRA delta
        h = A · h + B · e + trans_out                 # LTI-stable update (see below)
        p = sigmoid(halt(h))                          # ACT per-position halting probability
        # ACT remainder trick: if cumulative_p + p ≥ threshold, emit (1 - cumulative_p) as weight
        # gate by still_running so each position contributes exactly once on its halting step
        h_out += weight · h
  ↓
  Coda: coda_layers × TransformerBlock (dense SwiGLU FFN, no MoE)
  ↓ RMSNorm → LM head (weight-tied with embedding) → logits

Autoregressive generation uses KV caching with a separate cache key per loop depth (`recurrent_loop_{t}`) so every loop at every decode step finds populated keys.

Non-negotiable invariants — if you break these, the model breaks

1. **`ρ(A) < 1` always.** The entire reason `LTIInjection` exists is to guarantee this by construction. `A = exp(-exp(log_dt + log_A))` sits element-wise in (0, 1). Never replace this with a free parameter, never initialize `A` as a raw `nn.Parameter` of shape `(dim,)`, never remove the `clamp(-20, 20)` — that clamp exists so `log_dt → -∞, log_A → +∞` doesn't produce `0 · inf = NaN`. If the user sees spectral-radius drift or residual explosion, this is the first thing to check. 2. **`e` is frozen across loops.** `e` is set once after the Prelude and re-injected at every loop iteration. This is what prevents drift across arbitrary recurrence depth. If someone accidentally recomputes `e` inside the loop, they have silently changed the archite