/OpenMythos-Skill

name: openmythos
description: "Use when working with the OpenMythos codebase (Kye Gomez's PyTorch Recurrent-Depth Transformer): open_mythos/main.py, MythosConfig, OpenMythos, RecurrentBlock, LTIInjection, ACTHalting, MoEFFN, MLAttention, GQAttention, LoRAAdapter, mythos_1b..mythos_1t variants, Prelude/Recurrent/Coda layout, h=A*h+B*e+Transformer(h,e) update, loop-index embedding, ACT remainder trick, FineWeb-Edu looped training, or looped-training symptoms (loss spikes at a reproducible step, residual explosion, overthinking). Also when asked for 'Mythos mode'. A filename or symbol match is enough. Not for generic transformer, MoE or ACT questions."

OpenMythos

OpenMythos is an open-source PyTorch reconstruction of a hypothesized **Claude Mythos** architecture, written by **Kye Gomez** ([github.com/kyegomez/OpenMythos](https://github.com/kyegomez/OpenMythos), MIT license). It implements a **Recurrent-Depth Transformer (RDT)** with three stages — **Prelude** (standard transformer blocks, run once), a **Recurrent Block** (one TransformerBlock looped up to `max_loop_iters` times with input injection at every step), and a **Coda** (standard transformer blocks, run once). Attention is switchable between GQA and MLA; the FFN inside the recurrent block is a fine-grained MoE with always-on shared experts.

The project is an *independent, theoretical* reconstruction. It is not affiliated with Anthropic. The README is careful with language like "suspected", "likely", and "most probable class of solution", and so is this skill — don't claim this is what Anthropic actually does internally. If the user conflates OpenMythos with real Claude internals, gently correct them.

This skill turns Claude into a careful senior engineer who knows this specific repo. That's the entire job. (There is an optional experimental appendix at the bottom for users who want Claude to roleplay reasoning in the RDT's Prelude → Loop → Coda shape, but it is off by default.)

The repo at a glance

open_mythos/
├── main.py         — MythosConfig, OpenMythos, all nn.Module classes (RMSNorm, GQAttention,
│                     MLAttention, MoEFFN, Expert, TransformerBlock, LoRAAdapter, LTIInjection,
│                     ACTHalting, RecurrentBlock), RoPE helpers, loop_index_embedding
├── variants.py     — mythos_1b / 3b / 10b / 50b / 100b / 500b / 1t preset configs
├── tokenizer.py    — MythosTokenizer wrapper (defaults to openai/gpt-oss-20b via HF)
├── moda.py         — SEPARATE model: Mixture-of-Depths Attention + DeepSeek MoE (MoDAConfig,
│                     MoDAModel, own RMSNorm/RoPE). Not the looped RDT — the invariants below
│                     apply to main.py, not to this file.
└── __init__.py     — public re-exports

training/3b_fine_web_edu.py  — reference training script (FSDP under torchrun, FineWeb-Edu)
tests/                        — test_main.py, test_tokenizer.py, bench_vs_transformer.py,
                                small_benchmark.py, test_rope_debug.py
docs/                         — open_mythos.md (full class reference), datasets.md
examples/                     — moda_example.py, variants_example.py
example.py                    — minimal end-to-end sanity script at repo root

The forward pass — hold this in your head

input_ids
  ↓ embed
  ↓ Prelude: prelude_layers × TransformerBlock (dense SwiGLU FFN, no MoE)
  e = x  ← encoded input is frozen here, re-injected every loop
  ↓
  RecurrentBlock (one block, looped up to n_loops times; uses MoE FFN):
    for t in range(n_loops):
        h_loop = loop_index_embedding(h, t, dim//8)   # RoPE-like signal on a slice of channels
        combined = RMSNorm(h_loop + e)                 # input injection into normed stream
        trans_out = TransformerBlock(combined) + LoRAAdapter(trans_out, t)   # per-depth LoRA delta
        h = A · h + B · e + trans_out                 # LTI-stable update (see below)
        p = sigmoid(halt(h))                          # ACT per-position halting probability
        # ACT remainder trick: if cumulative_p + p ≥ threshold, emit (1 - cumulative_p) as weight
        # gate by still_running so each position contributes exactly once on its halting step
        h_out += weight · h
  ↓
  Coda: coda_layers × TransformerBlock (dense SwiGLU FFN, no MoE)
  ↓ RMSNorm → LM head (weight-tied with embedding) → logits

Autoregressive generation uses KV caching with a separate cache key per loop depth (`recurrent_loop_{t}`) so every loop at every decode step finds populated keys.

Non-negotiable invariants — if you break these, the model breaks

1. **`ρ(A) < 1` always.** The entire reason `LTIInjection` exists is to guarantee this by construction. `get_A()` returns `A = exp(-exp((log_dt + log_A).clamp(-20, 20)))`, a 1-D `(dim,)` tensor that sits element-wise in (0, 1), so `ρ(A) = A.max()`. Never replace this with a free parameter, never initialize `A` as a raw `nn.Parameter` of shape `(dim,)`, never remove the `clamp(-20, 20)` — that clamp exists so `log_dt → -∞, log_A → +∞` doesn't produce `0 · inf = NaN`. If the user sees spectral-radius drift or residual explosion, this is the first thing to check.

• **Reading `A.max()` correctly.** In exact arithmetic `A < 1` strictly, but floats round: at the clamp floor `A = 1 - 2e-9`, which fp32 rounds to exactly `1.0` once `log_dt + log_A < -17.3` (just above that, `A` is `1 - 6e-8`); in bf16 anything above ~0.998 (`log_dt + log_A < -6.2`) rounds to `1.0`. So a reading of `1.0000` does **not** by itself prove the module was modified — unmodified code can reach it when training drives a channel's decay toward zero. Recompute in fp32 and look at `s = (log_dt + log_A)`: `s.min()` near -20 means the channel is saturated at the clamp (and the clamp has zero gradient there, so it cannot recover). Only call the parameterization "broken" if the source differs from the form above (clamp removed, raw `Parameter`, `A` clipped by hand) or `A` is NaN / outside (0, 1).

2. **`e