This document describes how to convert a standard Gemma 4 model into a native FerrisRes Block AttnRes model through structural linearization — a lossless distillation process that preserves the model's capabilities while reducing attention complexity from O(n²) to O(n).
- Architecture Overview
- How It Works
- Prerequisites
- Quick Start
- CLI Reference
- Memory Requirements
- The Distillation Process
- Monitoring & Evaluation
- Troubleshooting
- Post-Distillation
- Technical Deep Dive
Standard transformer models like Gemma 4 use quadratic attention:
Attention cost: O(n²) — every token attends to every other token
KV Cache: Grows linearly with sequence length
At 100K tokens: 10 billion attention operations
Even with FlashAttention and PagedAttention, the fundamental scaling is quadratic. A 131K-token context requires prohibitive memory and compute.
FerrisRes implements a two-level attention hierarchy:
Level 1: Intra-block attention — tokens within a block attend locally
Level 2: Inter-block attention — block summaries attend globally
Result: O(n) scaling — constant work per token, regardless of context length
The key innovation is the Block Summary Layer that sits between these two levels. It compresses a block of tokens (e.g., 4096) into a single summary vector using cross-attention:
Block Tokens [4096 × hidden_dim]
↓ Cross-Attention
↓ with learnable summary queries
Summary [1 × hidden_dim]
At initialization, this layer is an identity transform (bridge_weight = 0), meaning the model behaves exactly like standard Gemma 4. During distillation, the layer gradually learns to compress information.
We don't train from scratch. We map Gemma 4's existing weights directly:
Gemma 4 Component → FerrisRes Component
───────────────────────────── ─────────────────────
Q/K/V/O projections → Block AttnRes attention weights
MoE expert weights (128×) → CpuMoELayer (preserved exactly)
RMSNorm weights → RMSNorm weights (direct copy)
Embedding / LM Head → Embedding / LM Head (may be tied)
[NEW] Block Summary queries → Identity init (zeros)
[NEW] Bridge weight → 0.0 (pass-through)
Result: At step 0, the student model produces identical output to the teacher. No accuracy is lost.
We then train only the Block Summary parameters:
Frozen: Attention weights (Q/K/V/O)
Frozen: MoE expert weights
Frozen: FFN weights
Frozen: Embedding / LM Head
Trainable: Block Summary queries (cross-attention)
Trainable: Bridge weight (0 → learned value)
Trainable: Output projection for summaries
Using KL divergence:
Loss = KL(P_teacher || P_student)
This requires only ~50,000 tokens of high-quality text — the model already knows how to process language; we're just teaching it a new way to compress context.
| Model | Min RAM | Min VRAM | Recommended |
|---|---|---|---|
| E2B | 8 GB | 4 GB | 16 GB RAM (CPU mode) |
| E4B | 16 GB | 8 GB | 24 GB RAM |
| 12B | 32 GB | 24 GB | 48 GB + GPU |
| 27B | 64 GB | 48 GB | Multi-GPU |
# Nix (recommended)
nix develop
# Or manual: Rust 1.75+, Vulkan SDK, wgpu-compatible GPU driverDownload from HuggingFace:
# E2B (recommended for testing — ~4 GB)
wget https://huggingface.co/google/gemma-4-e2b-it/resolve/main/model.safetensors
# E4B (small MoE — ~8 GB)
wget https://huggingface.co/google/gemma-4-e4b-it/resolve/main/model.safetensorscargo run -- infoThis shows your GPU, VRAM, and compute capabilities.
# E2B with default settings (1000 steps)
cargo run -- distill \
--model-path ./model.safetensors \
--config e2b
# With custom parameters
cargo run -- distill \
--model-path ./model.safetensors \
--config e2b \
--steps 5000 \
--seq-len 1024 \
--lr 0.00005 \
--temperature 2.0 \
--data training_data.txt \
--output my_distilled_model \
--log-every 50
# Using GGUF format (smaller download, quantized weights)
cargo run -- distill \
--model-path ./model-Q4_K_M.gguf \
--config e2b \
--model-format gguf \
--steps 1000
# With real tokenizer for accurate perplexity
cargo run -- distill \
--model-path ./model.safetensors \
--config e2b \
--tokenizer ./tokenizer.json \
--data training_data.txt
# Resume a previously interrupted run
cargo run -- distill \
--model-path ./model.safetensors \
--config e2b \
--resume my_distilled_model.checkpoint.bin \
--steps 2000
# Non-Gemma models (LLaMA, Mistral, Phi, Qwen)
cargo run -- distill \
--model-path ./llama-3.1-8b.safetensors \
--config llama3-8b \
--steps 1000
cargo run -- distill \
--model-path ./mistral-7b.gguf \
--config mistral-7b \
--model-format gguf# Watch the loss curve
# CSV now includes cosine similarity between teacher/student hidden states
cat my_distilled_model.loss_curve.csv
# step,kl_loss,bridge_weight,learning_rate,cosine_sim_avg
# 0,2.3456,0.0000,0.00001,0.9998
# 10,2.1234,0.0012,0.00001,0.9985
# ...
# Plot loss + cosine similarity
python3 -c "
import csv
import matplotlib.pyplot as plt
with open('my_distilled_model.loss_curve.csv') as f:
reader = csv.DictReader(f)
steps, losses, cosines = [], [], []
for row in reader:
steps.append(int(row['step']))
losses.append(float(row['kl_loss']))
c = row.get('cosine_sim_avg', 'n/a')
cosines.append(float(c) if c != 'n/a' else None)
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8))
ax1.plot(steps, losses)
ax1.set_ylabel('KL Divergence Loss')
ax1.set_title('Distillation Progress')
valid_cos = [(s, c) for s, c in zip(steps, cosines) if c is not None]
if valid_cos:
ax2.plot([s for s, _ in valid_cos], [c for _, c in valid_cos])
ax2.set_ylabel('Cosine Similarity')
ax2.set_xlabel('Step')
plt.savefig('loss_curve.png')
"# Teacher baseline
cargo run -- evaluate \
--model-path ./model.safetensors \
--config e2b \
--text "The meaning of life is"Usage: ferrisres distill [OPTIONS]
Required:
--model-path <PATH> Path to model weights (safetensors or GGUF)
Options:
--config <CONFIG> Model config [default: e2b]
Gemma: e2b, e4b, 12b, 27b, 27b-mm
LLaMA: llama3-8b, llama3-70b
Mistral: mistral-7b, mixtral-8x7b
Phi: phi3-mini
Qwen: qwen2-7b
--model-format <FMT> File format: safetensors or gguf [default: safetensors]
--seq-len <N> Training sequence length [default: 512]
--steps <N> Number of distillation steps [default: 1000]
--lr <RATE> Learning rate [default: 0.0001]
--temperature <T> KL divergence temperature [default: 2.0]
--data <PATH> Training text file (plain text)
--tokenizer <PATH> Path to tokenizer.json (HuggingFace format)
--output <PATH> Output prefix [default: distilled_model]
--log-every <N> Log every N steps [default: 1]
--resume <PATH> Resume from checkpoint file
--checkpoint-every <N> Save checkpoint every N steps [default: 100]
--converge <FRAC> Auto-stop when loss doesn't improve by this fraction
for --converge-patience steps. 0 = disabled [default: 0]
--converge-patience <N> Steps with no improvement before stopping [default: 50]
se GPU for matmul acceleration
Usage: ferrisres evaluate --model-path <PATH> --text <TEXT>
Options:
--config <CONFIG> Model config [default: e2b]
Usage: ferrisres info
Shows: GPU adapter, VRAM, compute capabilities, recommended settings.
For Gemma 4 E2B (hidden_dim=2048, 18 layers, vocab=256000):
Component Parameters FP16 Size QLoRA (NF4) Size
───────────────────── ────────── ───────── ────────────────
Embedding 524,288K ~1.0 GB ~250 MB
18 × Attention (QKVO) 18 × 4×4M ~288 MB ~72 MB
18 × FFN 18 × 50M ~1.8 GB ~450 MB
LM Head 524,288K ~1.0 GB ~250 MB
─────────────────────────────────────────────────────────────────
Total ~2B ~4.1 GB ~1.0 GB
Distillation overhead:
Block Summary params ~8K per layer ~16 KB (negligible)
Optimizer state (Adam) ~24K per layer ~48 KB (negligible)
With QLoRA + gradient offloading:
# This fits on 16 GB by using NF4 for frozen weights
cargo run -- distill \
--model-path ./model.safetensors \
--config e2b \
--seq-len 256 \
--steps 2000 \
--lr 0.00005The teacher runs in NF4 (~1 GB), student also in NF4 (~1 GB), with gradients only for Block Summary (~negligible). Total peak: ~3 GB.
Step N:
1. Sample a batch of tokens from training data
2. Teacher forward pass (frozen Gemma 4):
tokens → embedding → 18×(attn → FFN) → norm → lm_head → logits_teacher
3. Student forward pass (with Block Summary):
tokens → embedding → 18×(attn → [Block Summary?] → FFN) → norm → lm_head → logits_student
4. Compute KL divergence:
loss = Σ P_teacher[i] × log(P_teacher[i] / P_student[i])
5. Backprop through Block Summary only:
dL/d(bridge_weight), dL/d(queries), dL/d(projections)
6. Adam optimizer step on trainable params
7. Log: step, loss, bridge_weight, learning_rate
Expected loss curve for E2B:
Step KL Loss Bridge Weight Notes
──── ──────── ──────────── ─────
0 2.3-3.0 0.000 Identity — student = teacher
50 1.8-2.2 0.01-0.05 Learning to compress
200 0.8-1.2 0.10-0.30 Significant compression
500 0.3-0.6 0.20-0.50 Approaching convergence
1000 0.1-0.3 0.30-0.60 Converged
5000 0.05-0.15 0.40-0.70 Well-converged
The bridge_weight should increase monotonically (Adam clamp to [0, 1]). If it stays near 0, the Block Summary isn't learning — reduce temperature or increase learning rate.
Higher temperature = softer distributions = easier distillation:
Temperature 1.0: Sharp distributions, high KL initial loss
Temperature 2.0: Moderate softening (recommended default)
Temperature 4.0: Very soft, may over-smooth the signal
Temperature 8.0: Nearly uniform — too soft, don't use
- KL Divergence Loss — primary metric, should decrease monotonically
- Bridge Weight — should increase from 0 toward 0.3-0.7
- Cosine Similarity — layer-by-layer teacher vs student alignment (1.0 = perfect)
- Perplexity — teacher and student should converge
- Learning Rate — verify warmup is working
The CSV cosine_sim_avg column shows the average cosine similarity across
all layers between teacher and student hidden states. This is the most
informative distillation quality signal:
cosine_sim = 1.0 → Teacher and student are identical
cosine_sim = 0.99 → Nearly identical (excellent distillation)
cosine_sim = 0.95 → Minor divergence (good)
cosine_sim = 0.90 → Moderate divergence (acceptable)
cosine_sim = 0.80 → Significant divergence (check learning rate)
cosine_sim < 0.70 → Poor — something is wrong
Cosine similarity is computed every 10 steps (to save compute) and
logged as n/a on other steps. It measures direction alignment of the
hidden state vectors, independent of magnitude.
bridge_weight = 0.0 → Student ≈ Teacher (minimal compression)
bridge_weight = 0.1 → 10% summary (initial value)
bridge_weight = 0.3 → 30% summary, 70% raw hidden state
bridge_weight = 0.5 → Balanced compression
bridge_weight = 0.7 → Heavy compression (aggressive O(n))
bridge_weight = 1.0 → Fully compressed (maximum compression)
Values between 0.3–0.6 are typical for well-distilled models.
| Symptom | Cause | Fix |
|---|---|---|
| Loss doesn't decrease | LR too small | Increase --lr to 0.001 |
| Loss oscillates wildly | LR too large | Decrease --lr to 0.00001 |
| Bridge weight stuck at 0 | Gradients too small | Increase --temperature or --lr |
| Loss = 0 immediately | Bug in data loading | Check --data path |
| NaN in loss | Numerical instability | Reduce --temperature to 1.0 |
Error: Layer count mismatch: config says 18, safetensors has 48
Fix: You're using the wrong --config. Check the model name:
gemma-4-e2b→--config e2b(18 layers)gemma-4-e4b→--config e4b(24 layers)gemma-4-12b→--config 12b(48 layers)llama-3.1-8b→--config llama3-8b(32 layers)mistral-7b→--config mistral-7b(32 layers)phi-3-mini→--config phi3-mini(32 layers)
Error: Invalid GGUF magic
Fix: Make sure you're using --model-format gguf:
cargo run -- distill --model-path ./model.gguf --config e2b --model-format ggufGGUF files use different tensor naming (blk.N.attn_q.weight) than
safetensors (model.layers.N.self_attn.q_proj.weight). The GGUF loader
automatically maps these via standard_name_map(). If tensors are missing,
check the GGUF file's tensor names:
# Inspect GGUF metadata
python3 -c "
from gguf import GGUFReader
reader = GGUFReader('model.gguf')
for tensor in reader.tensors:
print(tensor.name)
"Error: Failed to read tokenizer.json
Fix: Download the tokenizer from HuggingFace:
wget https://huggingface.co/google/gemma-4-e2b-it/resolve/main/tokenizer.jsonWithout --tokenizer, the distillation uses byte-level tokenization.
This is fine for pipeline testing but produces meaningless perplexity.
For real distillation, always provide a tokenizer.
"Hidden dim mismatch"
Error: Hidden dim mismatch: config says 2048, safetensors has 4608
Fix: Wrong config again. Use --config e2b for E2B models only.
AllocationError: Out of memory
Fix: Reduce --seq-len (try 256 or 128) or use a smaller model.
This is a warning, not an error. Without --data, the distillation uses
synthetic tokens (0, 1, 2, ...). This is fine for testing the pipeline
but won't produce meaningful results. Use real data for production runs.
| Dataset | Size | Quality | Link |
|---|---|---|---|
| Wikitext-103 | 500 MB | High | huggingface/datasets/wikitext |
| C4 (subset) | 1-10 GB | High | huggingface/datasets/c4 |
| OpenWebText | 12 GB | High | huggingface/datasets/openwebtext |
| RedPajama | 1 TB | Mixed | togethercomputer/RedPajama |
For distillation, you only need ~50K tokens (a few MB of text). Quality matters more than quantity.
After distillation, the output includes:
distilled_model.bin.block_summary_0.bin # Trained Block Summary params
distilled_model.bin.checkpoint.bin # Full checkpoint (for resume)
distilled_model.bin.loss_curve.csv # Loss + cosine similarity history
If a distillation run is interrupted (OOM, Ctrl+C, crash), resume from the last checkpoint:
cargo run -- distill \
--model-path ./model.safetensors \
--config e2b \
--resume distilled_model.checkpoint.bin \
--steps 2000 # Additional steps from checkpointThe --steps value is additive — if the checkpoint was at step 100,
--steps 2000 runs steps 100 through 2100.
The checkpoint preserves:
- Block Summary trainable parameters
- Adam optimizer state (first moment m, second moment v, timestep t)
- Bridge weight values
- Layer norm weights
- Global step counter
Use --converge to stop training automatically when loss plateaus:
cargo run -- distill \
--model-path ./model.safetensors \
--config 27b-mm \
--steps 10000 \
--converge 0.001 \
--converge-patience 100 \
This stops when the KL loss hasn't improved by 0.1% for 100 consecutive
steps. You'll see event="converged" in the structured logs. Set
--converge 0 (default) to disable.
The distilled model can be loaded for O(n) inference:
use ferrisres::model::gemma_mapper::{
load_gemma4_model, Gemma4Config, Gemma4Student,
BlockSummaryLayer, DistillationConfig,
DistillationCheckpoint,
};
// Load model
let config = Gemma4Config::gemma4_e2b();
let model = load_gemma4_model(path, config)?;
// Load checkpoint to restore Block Summary params
let ckpt = DistillationCheckpoint::load("distilled_model.checkpoint.bin")?;
let block_summaries = /* restored from ckpt */;
// Or save params separately with the safetensors writer
use ferrisres::model::safetensors::{write_safetensors, TensorToWrite, SafeDtype};
write_safetensors(
Path::new("distilled_block_summaries.safetensors"),
&tensor_list,
)?;
// Run inference — O(n) instead of O(n²)
let student = Gemma4Student::new(model, block_summaries, config);
let logits = student.forward(&token_ids);After distillation, the student model should have:
| Metric | Before | After |
|---|---|---|
| Attention complexity | O(n²) | O(n) |
| KV Cache (per token) | hidden_dim × 2 | ~hidden_dim / 16 |
| Max context length | 8K-32K (practical) | 131K+ (linear) |
| Quality vs teacher | 100% | 95-99% |
| Inference speed (short) | Baseline | ~1.1× (overhead) |
| Inference speed (long) | O(n²) | O(n) — much faster |
The 1-5% quality loss is the trade-off for O(n) scaling. At very long contexts (32K+), the speed improvement far outweighs the minor quality reduction.
The core mechanism:
Input: Block tokens [block_size × hidden_dim]
Query: Learnable summary queries [num_queries × hidden_dim]
Output: Summary [num_queries × hidden_dim]
1. Project queries: Q = queries × W_q
2. Project keys: K = tokens × W_k
3. Project values: V = tokens × W_v
4. Attention: A = softmax(Q × K^T / √d) × V
5. Output: S = A × W_o
6. Bridge: output = (1-w) × mean_pool(tokens) + w × S
The bridge weight w controls compression aggressiveness:
w=0: No compression, student = teacherw=1: Full compression, O(n) but maximal information loss
At initialization, all Block Summary parameters are set to produce an identity mapping:
summary_queries = vec![0.0; num_queries × hidden_dim]; // Zero → mean pool
query_proj = identity_matrix; // Pass-through
key_proj = identity_matrix;
value_proj = identity_matrix;
out_proj = identity_matrix;
bridge_weight = 0.1; // Small initial contributionThis ensures that at step 0, the student produces exactly the same output as the teacher. The distillation only adjusts the compression, never degrading what the model already knows.
During backpropagation, gradients flow only through the Block Summary:
Loss → d_logits → d_hidden → d_block_summary_output
↓
┌─────────┴──────────┐
↓ ↓
d_bridge_weight d_summary_queries
↓ ↓
Adam update Adam update
All other weights: FROZEN (no gradient)
This is implemented via the autodiff graph:
BlockSummaryCrossAttnnode inComputationGraph- Backward pass allocates gradient buffers for tokens and queries
- CPU-side
backprop_block_summary()computes exact gradients BlockSummaryAdamapplies Adam with bias correction
The weight loader expects standard HuggingFace naming:
model.embed_tokens.weight # [vocab × hidden]
model.layers.N.self_attn.{q,k,v,o}_proj.weight # [hidden × hidden]
model.layers.N.input_layernorm.weight # [hidden]
model.layers.N.post_attention_layernorm.weight # [hidden]
model.layers.N.mlp.{gate,up,down}_proj.weight # Dense FFN
model.layers.N.block_sparse_moe.gate.weight # MoE router
model.layers.N.block_sparse_moe.experts.E.w1.weight # Expert gate
model.layers.N.block_sparse_moe.experts.E.w3.weight # Expert up
model.layers.N.block_sparse_moe.experts.E.w2.weight # Expert down
model.norm.weight # Final norm
lm_head.weight # [vocab × hidden]
Gemma 4:
| Parameter | E2B | E4B | 12B | 27B | 27B MM |
|---|---|---|---|---|---|
| hidden_dim | 2048 | 2560 | 4096 | 4608 | 1536 |
| num_layers | 18 | 24 | 48 | 60 | 35 |
| num_heads | 8 | 10 | 32 | 36 | 8 |
| num_kv_heads | 1 | 1 | 8 | 8 | 1 |
| head_dim | 256 | 256 | 128 | 128 | 192 |
| intermediate_dim | 8192 | 10240 | 14336 | 16384 | 6144 |
| num_experts | 1 (dense) | 16 | 128 | 128 | 1 (dense) |
| top_k | 1 | 2 | 2 | 2 | 1 |
| vocab_size | 256K | 256K | 256K | 256K | 262K |
| max_positions | 32K | 32K | 131K | 131K | 32K |
| sliding_window | 4096 | 4096 | 4096 | 4096 | 4096 |
| MoE layers | none | 6,12,18 | every other | every other | 4,9,14,19,24,29,34 |
| FP16 size | ~4 GB | ~8 GB | ~24 GB | ~54 GB | ~10 GB |
Other supported architectures:
| Model | --config |
hidden | layers | params | Notes |
|---|---|---|---|---|---|
| LLaMA 3.1 8B | llama3-8b |
4096 | 32 | ~8B | Dense |
| LLaMA 3.1 70B | llama3-70b |
8192 | 80 | ~70B | Dense |
| Mistral 7B | mistral-7b |
4096 | 32 | ~7B | Dense |
| Mixtral 8x7B | mixtral-8x7b |
4096 | 32 | ~47B | 8 experts |
| Phi-3 Mini | phi3-mini |
3072 | 32 | ~3.8B | Dense |
| Qwen 2.5 7B | qwen2-7b |
3584 | 28 | ~7B | Dense |
All models use the same RMSNorm + GQA + SwiGLU architecture and are compatible with both safetensors and GGUF loading.
| File | Purpose |
|---|---|
src/model/gemma_mapper.rs |
Core distillation logic (~3500 LOC) |
src/model/safetensors.rs |
Weight loading + safetensors writer |
src/model/gguf.rs |
GGUF weight loading + dequantization |
src/model/tokenizer.rs |
SimpleTokenizer, BpeTokenizer, HfTokenizer |
src/autodiff/graph.rs |
BlockSummaryCrossAttn node |
src/autodiff/backward.rs |
Backward pass for Block Summary |
src/main.rs |
CLI entry points |
Q: Can I use this with non-Gemma models (LLaMA, Mistral)?
A: Yes! The pipeline now supports LLaMA 3, Mistral, Mixtral, Phi-3, and
Qwen 2 out of the box. Use --config llama3-8b, --config mistral-7b,
--config phi3-mini, or --config qwen2-7b. All share the same
RMSNorm + GQA + SwiGLU architecture.
Q: Can I use GGUF files instead of safetensors?
A: Yes. Use --model-format gguf and point to any GGUF file. The loader
supports Q4_0, Q8_0, Q4_K, Q5_K, Q6_K, Q2_K, Q3_K quantization formats,
automatically dequantizing to F32 for the teacher/student forward pass.
GGUF files are typically 2-4× smaller than FP16 safetensors.
Q: How long does distillation take? A: E2B on CPU: ~30-60 minutes for 1000 steps. With GPU: ~5-10 minutes. 12B on CPU: several hours. Multi-GPU: ~30-60 minutes.
Q: What if the loss doesn't converge? A: The most common issue is learning rate. Start with 0.0001, try 0.0005 if it's too slow, or 0.00001 if it oscillates.
Q: Can I resume a distillation run?
A: Yes. Use --resume <checkpoint.bin> to restore the full training state
including Adam optimizer moments and Block Summary parameters. The
checkpoint is saved automatically at the end of each run.
Q: Do I need a tokenizer?
A: For testing the pipeline, no — it uses byte-level fallback. For real
distillation with meaningful perplexity, yes — use --tokenizer tokenizer.json
with a HuggingFace tokenizer file. Without it, perplexity numbers are
not comparable to published benchmarks.
Q: What's the minimum data needed? A: ~50K tokens (a few MB). Quality matters more than quantity — use Wikitext or C4, not random web text.
Q: Is the conversion truly lossless? A: At step 0: yes, identical to teacher. After distillation: 95-99% of teacher quality, measured by perplexity. The 1-5% loss is the trade for O(n) scaling.
Q: What does cosine similarity tell me? A: It measures how similar the teacher and student hidden states are at each layer. A value of 1.0 = identical, 0.99+ = excellent. If it drops below 0.90, the Block Summary is compressing too aggressively — reduce learning rate or increase temperature.
FerrisRes automatically detects and uses the GPU when available. No --gpu flag is needed.
cargo run -- distill --model-path ./model.safetensors --config 27b-mm --steps 100The GPU forward pass uses a profile-driven dispatch system:
- Queries
device.limits().max_buffer_sizeat startup - Per-layer weights are uploaded to GPU just-in-time for each matmul (~12ms/layer on PCIe 3.0)
- Large tensors (embed_tokens, lm_head) stay on CPU
- Scales from laptop iGPU (256MB buffers) to datacenter GPUs (multi-GB buffers)
- Hybrid CPU/GPU: GPU for linear projections (the bottleneck), CPU for attention/residuals/activations
| Hardware | Model | Config | Result |
|---|---|---|---|
| Lenovo X1 Yoga (16GB RAM, Intel UHD 620) | Gemma 4 27B MM | 27b-mm | ✅ Teacher forward ~3min, Student forward ~3min |
| (Any Vulkan GPU) | Gemma 4 E2B | e2b | ✅ Faster — smaller model |
After distillation, FerrisRes can continue improving through a closed-loop self-correction system:
- WASM Sandbox validates model-generated code in <1ms with zero host access
- LSP-as-Oracle provides deterministic compiler errors via rust-analyzer/pyright
- Mirror Test — model generates code, generates tests, executes tests
- Compiler errors → loss → backprop — the model is penalized at the weight level
- Concept Memory — learned patterns persist across sessions
This means the distillation doesn't stop at the initial training run. The model continues to refine itself through tool-mediated feedback.