Phase 4: gate reformulation falsified across both follow-on archs

Two architectures proposed in distractor_mix_README.md as paths to
make the HBit-tension attention gate earn its keep — both tested,
both falsified at distractor_frac=0.20:

  arch              mean    std    wins vs crt_only
  crt_only          2.4595  0.0257    —
  hybrid_score      2.5488  0.0239    0/3   (+3.6%)
  hybrid_learned    2.5607  0.0179    0/3   (+4.1%)

Combined with the original falsified key-gate (0/3, +3.2%), this is
THREE different gate formulations all losing to crt_only by the
same ~3-4% magnitude.

Interpretation (full writeup in GATE_REFORMULATION_RESULTS.md):
substrate-distance signal on (q@k^T / sqrt(d)) just doesn't
correlate with what the attention should focus on. The Fibonacci
attractor structure of OMC's HInt doesn't transfer to attention
score tensors which have totally different distributional
properties (Gaussian-ish, scaled, drawn from learned projections).

The learned-threshold variant is the stronger negative: the model
could simply learn gate_w≈0 to disable the gate. It did not. It
converged to a setting that costs ~4% on val loss.

Architectural verdict consolidated:
  - Substrate as positional encoding (CRT-PE):     WINS (-5.4%, -2.9%)
  - Substrate as OOD signal (HBit cross-cutting):  WINS (AUROC 1.0)
  - Substrate as attention modulator:              FALSIFIED 3×
                                                   (key, score, learned)

Recommended pivot: stop investing in attention-gate variants.
The next-most-promising substrate role is at the tokenizer /
embedding layer (using OMC's substrate-routed tokenizer as the
LM's input tokenizer), which is architecturally unmeasured and
has stronger theoretical grounding than another attention-layer
attempt.

Implementation:
  - models.py: Attention now takes gate_mode ∈ {none, key, score, learned}
  - hybrid_score: log_gate added to scores pre-mask, pre-softmax
  - hybrid_learned: per-block scalar W, b learnable; init W=-1, b=0
    so initial gate ≈ original 1/(1+d), then trains from loss signal
  - train_gate_reformulation.py: focused 3-arch comparison vs crt_only

Per-seed wall-clock ~10 min, total ~30 min on CPU.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Author: RandomCoder-lab

README.md

@@ -350,7 +350,7 @@ Submit a package: PR an entry to [`registry/index.json`](registry/index.json).
 | Harmonic libraries on real data | shipped, mixed-honest results |
 | Hybrid LLM experiments (12 experiments) | shipped, 1 perfect AUROC, 1 architectural negative, 1 CRT-PE win |
 | End-to-end transformerless LM (PyTorch) | CRT-PE wins -19.9% (tiny), **-5.4% (TinyShakespeare, 3/3 seeds), -2.9% (distractor mix, 3/3)** |
-| Hybrid HBit-gate distractor-mix test | **falsified at current gate formulation (0/3 wins)**, score-level / learned-threshold reformulations documented |
+| Hybrid HBit-gate distractor-mix test | **falsified across THREE gate formulations** (0/3 wins each, +3–4% consistent loss): KEY-magnitude gate, SCORE-level gate, LEARNED-threshold gate. The architectural pivot per [`GATE_REFORMULATION_RESULTS.md`](experiments/transformerless_lm/GATE_REFORMULATION_RESULTS.md): substrate's home is positional + distributional, not as an attention-score shaper. |
 | Self-hosting compiler V.9b | shipped, gen2 == gen3 byte-identical |
 | **Self-healing pass (7 classes, substrate-routed typo)** | shipped, `OMC_HEAL=1`, **10× typo lookup**, 16 tests, per-class pragmas |
 | **Substrate-keyed code codec + compressed messaging** | **shipped**, `omc_codec_encode/decode_lookup` + `omc_msg_sign_compressed/recover`, alpha-rename invariant, token-count ~N× (wire-byte breaks even at ≥500 B + N≥8); always-on win is library-lookup recovery; 13 tests, lossless on in-library content |

experiments/transformerless_lm/GATE_REFORMULATION_RESULTS.md

@@ -0,0 +1,152 @@
+# Gate reformulation — both follow-on architectures falsified
+
+## Context
+
+`distractor_mix_README.md` reported the original `hybrid` arch
+(CRT-PE + KEY-magnitude HBit gate) losing 0/3 to `crt_only` at
+distractor_frac=0.20. The writeup proposed two concrete
+reformulations that kept CRT-PE and changed only the gate:
+
+1. **`hybrid_score`** — gate on raw attention SCORES additively in
+   log-space pre-softmax, instead of post-softmax renormalization
+   on key magnitudes.
+2. **`hybrid_learned`** — replace fixed `1/(1+d)` with
+   `sigmoid(W*d + b)` where W, b are learned per-head. Lets the
+   model discover its own threshold and slope.
+
+Both kept CRT-PE intact. The hypothesis was that the original gate's
+formulation was too rigid, and a softer or learnable variant might
+earn its keep.
+
+## Setup
+
+Identical to `train_distractor_mix.py`:
+- TinyShakespeare, 90/10 split
+- 20% of training chunks char-shuffled (within-vocab distractors)
+- Validation on PURE shakespeare (the actual task we care about)
+- d_model=128, n_blocks=4, seq_len=128, ~801K params (+8 for learned gate)
+- 1500 steps, batch=32, AdamW lr=3e-4
+- 3 seeds: 42, 7, 123
+- CPU, ~30 min total wall-clock for 3 archs × 3 seeds
+
+## Results
+
+| arch | mean | std | wins vs crt_only | rel |
+|---|--:|--:|:-:|--:|
+| `crt_only` | **2.4595** | 0.0257 | — | — |
+| `hybrid_score` | 2.5488 | 0.0239 | **0/3** | **+3.6%** |
+| `hybrid_learned` | 2.5607 | 0.0179 | **0/3** | **+4.1%** |
+
+### Per-seed
+
+| seed | crt_only | hybrid_score | hybrid_learned |
+|---|--:|--:|--:|
+| 42 | 2.489 | 2.562 | 2.567 |
+| 7  | 2.443 | 2.521 | 2.540 |
+| 123| 2.446 | 2.564 | 2.574 |
+
+### Combined with the original
+
+| arch | mean | wins vs crt_only |
+|---|--:|:-:|
+| `crt_only` | 2.4595 | — |
+| `hybrid` (key-gate, original)     | 2.5379 | 0/3 |
+| `hybrid_score` (score-gate)       | 2.5488 | 0/3 |
+| `hybrid_learned` (learned-thresh) | 2.5607 | 0/3 |
+
+**Three different gate formulations, three falsifications. Same
++3-4% loss magnitude across all three.**
+
+## Interpretation
+
+The architectural read consolidates: **HBit tension is not a useful
+attention modulator at this scale and data regime**, regardless of
+where in the attention path the gate fires or whether its threshold
+is learnable.
+
+Why this is a stronger negative than the original single failure:
+- We tested two DIFFERENT failure modes the README proposed
+- The learned-threshold variant had every chance to recover —
+  the model could simply learn `gate_w ≈ 0, gate_b ≈ large` to
+  disable the gate entirely. It did not converge there; it
+  converged to a gate setting that costs ~4% on val loss.
+- The score-level variant operates at the correct layer of the
+  computation (logits, not key-magnitudes), removing the
+  "wrong-layer-of-abstraction" objection.
+
+This suggests the failure isn't a formulation bug — the underlying
+substrate-distance signal on `q@k^T / sqrt(d)` values just doesn't
+correlate with what the model needs to focus on. The Fibonacci
+attractor structure of OMC's `HInt` doesn't transfer to attention
+score tensors which have totally different distributional
+properties (Gaussian-ish, scaled by `1/sqrt(d_head)`, drawn from
+learned projections of token embeddings).
+
+## What this means for the transformerless LM
+
+The substrate's role in a transformer replacement is now empirically:
+
+| Component | Substrate variant | Status |
+|---|---|:-:|
+| Positional encoding | CRT-Fibonacci PE | **Wins** (−5.4% clean, −2.9% distractor mix; 3/3 + 4/5 seeds) |
+| OOD detection | HBit cross-cutting tension | **Wins** (AUROC 1.0 on scenario A) |
+| Attention modulation (key-mag gate) | `1/(1+d)` on `\|k\|.mean` | **Falsified** (0/3) |
+| Attention modulation (score-level gate) | `1/(1+d)` on logits pre-softmax | **Falsified** (0/3, this writeup) |
+| Attention modulation (learned threshold) | `sigmoid(W*d+b)` on `\|k\|.mean` | **Falsified** (0/3, this writeup) |
+
+**The substrate's home in this architecture is positional and
+distributional, not as an attention-score shaper.** Three independent
+attempts to make it work there have all failed by similar margins.
+
+## What's left to try (the new menu)
+
+Since attention-gate variants are exhausted at this scale, the
+remaining places to introduce substrate signal are all
+out-of-attention:
+
+### A. FFN substrate gate (vs attention gate)
+The FFN block doesn't have softmax — substrate signal there is
+unmediated. Apply `attractor_distance` to the post-GELU
+activations or to one of the linear projections. The FFN
+operates on per-position vectors with no cross-position coupling,
+so the substrate distance is computed in the same per-position
+basis OMC's HInt was designed for.
+
+### B. Auxiliary substrate loss (regularizer, not forward signal)
+Add `lambda * attractor_distance(activations).mean()` as an
+auxiliary loss term. Gradients pull the network toward
+substrate-aligned representations without affecting the forward
+pass. Closest analog: weight decay, but in attractor-distance
+space instead of L2.
+
+### C. Substrate-curriculum sampling (training order, not architecture)
+Sort training batches by attractor-distance of their token IDs:
+on-attractor samples first, off-attractor later. The substrate
+becomes a curriculum signal, not an architecture change. Cheap
+to test (no model change needed).
+
+### D. Per-head selective gating
+Some heads get the substrate gate, some don't. Train one of the
+existing falsified variants but applied to only 1-of-4 heads.
+This is the weakest "maybe" — if the gate fails on all heads it
+will likely fail on one too — but worth ruling out cleanly.
+
+### E. Honest pivot
+Accept that HBit-as-attention-gate is dead at the scales we can
+test. Ship the transformerless prototype with CRT-PE + standard
+softmax attention + substrate-aware tokenization (which is the
+biggest unexplored axis — the substrate at the EMBEDDING layer,
+not the attention layer). This is the path that respects what
+we've actually measured.
+
+## Recommendation
+
+E first, then A in parallel. Stop investing in attention-gate
+formulations — three failures with consistent magnitude is a
+saturation signal. The pivot toward substrate-aware tokenization
+hasn't been measured yet and has a stronger architectural basis
+(OMC's tokenizer is already substrate-routed; using it as the LM's
+input tokenizer is a small change with potentially large effect).
+
+Numbers taken on 2026-05-16. Same hardware as the original
+distractor-mix experiment. Per-seed wall-clock ~10 min for 3 archs.

experiments/transformerless_lm/models.py

@@ -109,47 +109,73 @@ def hbit_tension_gate(keys: torch.Tensor, scale: float = 1.0) -> torch.Tensor:
 
 
 class Attention(nn.Module):
-    """Single-head attention. Optionally wraps softmax with an HBit
-    tension gate computed from the (post-projection) key magnitudes.
-
-    The gate is computed on per-key SCALAR summary (mean over d_head).
-    This is the architectural mapping from our 1-D HBit-tension
-    experiment to a real attention layer. More sophisticated gates
-    (per-channel, learned threshold) are possible but the simplest
-    match to experiment 12 is per-key.
+    """Single-head attention. The gate_mode parameter selects how (if
+    at all) the HBit-tension signal modulates attention:
+
+      "none"     : pure softmax (standard / crt_only).
+      "key"      : fixed gate on per-key magnitude (the falsified
+                   distractor-mix formulation; kept for reference).
+      "score"    : ADDITIVE log-gate on the score tensor pre-softmax.
+                   Substrate-distance of the raw score values dampens
+                   off-attractor logits before softmax normalizes. No
+                   post-hoc renormalization needed.
+      "learned"  : per-head learnable scalar (W, b) gate on per-key
+                   magnitude. Initialized to approximate the fixed
+                   1/(1+d) formula; trains to discover whether
+                   substrate distance is a useful signal for the task.
     """
 
-    def __init__(self, d_model: int, use_hbit_gate: bool, dropout: float = 0.0):
+    def __init__(self, d_model: int, gate_mode: str = "none", dropout: float = 0.0):
         super().__init__()
+        if gate_mode not in ("none", "key", "score", "learned"):
+            raise ValueError(f"unknown gate_mode: {gate_mode}")
         self.d_model = d_model
         self.qkv = nn.Linear(d_model, 3 * d_model)
         self.out = nn.Linear(d_model, d_model)
-        self.use_hbit_gate = use_hbit_gate
+        self.gate_mode = gate_mode
         self.dropout = dropout
+        if gate_mode == "learned":
+            # Initialize so sigmoid(W*d + b) ≈ 1/(1 + d) near d ≈ 0:
+            # picking W = -1, b = 0 gives sigmoid(-d) ∈ (0, 0.5], a
+            # softer version of the falsified gate. Both are learnable.
+            self.gate_w = nn.Parameter(torch.tensor(-1.0))
+            self.gate_b = nn.Parameter(torch.tensor(0.0))
 
     def forward(self, x: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
-        # x: [B, T, D]
         B, T, D = x.shape
         qkv = self.qkv(x)
-        q, k, v = qkv.chunk(3, dim=-1)  # each [B, T, D]
+        q, k, v = qkv.chunk(3, dim=-1)
         scale = 1.0 / math.sqrt(D)
         scores = (q @ k.transpose(-2, -1)) * scale  # [B, T, T]
+
+        if self.gate_mode == "score":
+            # Gate on score VALUES (pre-mask). attractor_distance of
+            # raw scores tells us whether the (q·k) magnitude lands
+            # on a substrate attractor. Off-attractor scores get
+            # additively penalized in log-space, so softmax handles
+            # normalization natively.
+            d = attractor_distance(scores * 10.0)  # [B, T, T]
+            log_gate = -torch.log1p(d)
+            scores = scores + log_gate
+
         scores = scores.masked_fill(mask == 0, float('-inf'))
         attn = F.softmax(scores, dim=-1)  # [B, T, T]
 
-        if self.use_hbit_gate:
-            # gate per key: 1 scalar per key position (mean of |k|).
-            # Shape [B, T]. Apply along the key axis.
-            key_mag = k.abs().mean(dim=-1)  # [B, T]
-            gate = hbit_tension_gate(key_mag * 100.0)  # scale up so attractor_distance is meaningful
-            # broadcast gate over the query axis: [B, 1, T]
+        if self.gate_mode == "key":
+            key_mag = k.abs().mean(dim=-1)
+            gate = hbit_tension_gate(key_mag * 100.0)
+            attn = attn * gate.unsqueeze(1)
+            attn = attn / (attn.sum(dim=-1, keepdim=True) + 1e-9)
+        elif self.gate_mode == "learned":
+            key_mag = k.abs().mean(dim=-1)
+            d = attractor_distance(key_mag * 100.0)  # [B, T]
+            gate = torch.sigmoid(self.gate_w * d + self.gate_b)
             attn = attn * gate.unsqueeze(1)
-            # renormalize so attn rows still sum to ~1
             attn = attn / (attn.sum(dim=-1, keepdim=True) + 1e-9)
 
         if self.dropout > 0 and self.training:
             attn = F.dropout(attn, p=self.dropout)
-        out = attn @ v  # [B, T, D]
+        out = attn @ v
         return self.out(out)
 
 
@@ -172,9 +198,9 @@ def forward(self, x):
 
 
 class Block(nn.Module):
-    def __init__(self, d_model: int, use_hbit_gate: bool):
+    def __init__(self, d_model: int, gate_mode: str = "none"):
         super().__init__()
-        self.attn = Attention(d_model, use_hbit_gate=use_hbit_gate)
+        self.attn = Attention(d_model, gate_mode=gate_mode)
         self.ff = FeedForward(d_model)
         self.ln1 = nn.LayerNorm(d_model)
         self.ln2 = nn.LayerNorm(d_model)
@@ -186,8 +212,7 @@ def forward(self, x, mask):
 
 
 class TinyLM(nn.Module):
-    """Tiny char-level LM. Three architectures behind one class via
-    constructor flags."""
+    """Tiny char-level LM. Architecture selected via constructor flags."""
 
     def __init__(
         self,
@@ -196,7 +221,7 @@ def __init__(
         n_blocks: int,
         seq_len: int,
         pe_kind: str,             # "sinusoidal" or "crt"
-        use_hbit_gate: bool,
+        gate_mode: str,           # "none" | "key" | "score" | "learned"
     ):
         super().__init__()
         self.seq_len = seq_len
@@ -207,27 +232,24 @@ def __init__(
             pe = crt_pe(seq_len, d_model)
         else:
             raise ValueError(f"unknown pe_kind: {pe_kind}")
-        self.register_buffer("pe", pe)  # [seq_len, d_model]
+        self.register_buffer("pe", pe)
         self.blocks = nn.ModuleList([
-            Block(d_model, use_hbit_gate=use_hbit_gate) for _ in range(n_blocks)
+            Block(d_model, gate_mode=gate_mode) for _ in range(n_blocks)
         ])
         self.ln_f = nn.LayerNorm(d_model)
         self.head = nn.Linear(d_model, vocab_size, bias=False)
-        # tie head weights to embedding
         self.head.weight = self.embed.weight
-        # causal mask
         mask = torch.tril(torch.ones(seq_len, seq_len))
         self.register_buffer("mask", mask)
 
     def forward(self, x):
-        # x: [B, T]
         B, T = x.shape
         h = self.embed(x) + self.pe[:T]
         mask = self.mask[:T, :T]
         for block in self.blocks:
             h = block(h, mask)
         h = self.ln_f(h)
-        return self.head(h)  # [B, T, vocab]
+        return self.head(h)
 
 
 def make_model(
@@ -237,19 +259,27 @@ def make_model(
     d_model: int = 64,
     n_blocks: int = 2,
 ) -> TinyLM:
-    """Convenience: build one of the three benchmarked architectures.
-    Defaults match the original tiny-bench (d_model=64, n_blocks=2).
-    The scale experiment uses d_model=128, n_blocks=4."""
+    """Five architectures:
+      standard       : sinusoidal PE + pure softmax
+      crt_only       : CRT-Fib PE   + pure softmax
+      hybrid         : CRT-Fib PE   + KEY-magnitude gate (falsified)
+      hybrid_score   : CRT-Fib PE   + SCORE-level pre-softmax gate
+      hybrid_learned : CRT-Fib PE   + LEARNED per-head gate on key magnitude
+    """
     common = dict(
         vocab_size=vocab_size,
         d_model=d_model,
         n_blocks=n_blocks,
         seq_len=seq_len,
     )
     if arch == "standard":
-        return TinyLM(**common, pe_kind="sinusoidal", use_hbit_gate=False)
+        return TinyLM(**common, pe_kind="sinusoidal", gate_mode="none")
     if arch == "crt_only":
-        return TinyLM(**common, pe_kind="crt", use_hbit_gate=False)
+        return TinyLM(**common, pe_kind="crt", gate_mode="none")
     if arch == "hybrid":
-        return TinyLM(**common, pe_kind="crt", use_hbit_gate=True)
+        return TinyLM(**common, pe_kind="crt", gate_mode="key")
+    if arch == "hybrid_score":
+        return TinyLM(**common, pe_kind="crt", gate_mode="score")
+    if arch == "hybrid_learned":
+        return TinyLM(**common, pe_kind="crt", gate_mode="learned")
     raise ValueError(f"unknown arch: {arch}")