Skip to content

Commit e4a1839

Browse files
committed
Update STEERING_RESULTS.md for consistency with Code-LLaMA and Phi-3 findings
Add code-specialization context and scope notes throughout, reference cross-model attention analysis in RIGOR_EXPERIMENT.md, add LLaMA and Phi-3 backends to files table, expand future work with steering on non-code-specialized models, and add footer.
1 parent 93d3c77 commit e4a1839

1 file changed

Lines changed: 22 additions & 6 deletions

File tree

docs/experiments/STEERING_RESULTS.md

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,22 @@
66

77
## Executive Summary
88

9-
We successfully implemented and validated attention steering in PLIP-rs, demonstrating that:
9+
We successfully implemented and validated attention steering in PLIP-rs on the two Qwen code-specialized models, demonstrating that:
1010

11-
1. **Rust test markers receive significantly less attention than Python doctest markers** (2.3-2.6% vs 5.7-9.1%)
11+
1. **Rust test markers receive significantly less attention than Python doctest markers** (2.3-2.6% vs 5.7-9.1% in Qwen models; 2.8-4.4× ratio across all 4 code-specialized models — see [RIGOR_EXPERIMENT.md](RIGOR_EXPERIMENT.md))
1212
2. **Steering can boost Rust attention to Python levels** without catastrophically affecting model outputs
1313
3. **The intervention is "safe"** - KL divergence remains flat across dose levels
1414

15+
**Scope**: Steering dose-response experiments were conducted on Qwen-3B and Qwen-7B only. The attention asymmetry that motivates steering has since been validated across 4 code-specialized models but does **not replicate** on Code-LLaMA-7B (reversed pattern) or Phi-3-mini (no significant difference). See [RIGOR_EXPERIMENT.md](RIGOR_EXPERIMENT.md) Appendix C for full cross-model analysis.
16+
1517
---
1618

1719
## 1. Calibration Results
1820

1921
### 1.1 Baseline Attention Measurements
2022

23+
Calibration was performed on the two Qwen code-specialized models. For cross-model attention analysis across all 6 models (including Code-LLaMA-7B and Phi-3-mini), see [RIGOR_EXPERIMENT.md](RIGOR_EXPERIMENT.md) Appendix C.
24+
2125
| Model | Layer | Python `>>>``fn` | Rust `#[test]``fn` | Ratio |
2226
|-------|-------|---------------------|----------------------|-------|
2327
| **Qwen/Qwen2.5-Coder-3B-Instruct** | 20 | 5.70% | 2.32% | 2.46× |
@@ -142,6 +146,8 @@ Both Qwen models exhibit consistent attention asymmetry:
142146
- **Rust `#[test]` markers**: Receive only 2.3-2.6% attention to function tokens
143147
- **Gap widens with model size**: 2.46× ratio for 3B → 3.51× ratio for 7B
144148

149+
This asymmetry has been confirmed across all 4 code-specialized models (Qwen-7B/3B, StarCoder2-3B, CodeGemma-7B) with ratios of 2.8-4.4× and p < 0.0002. However, two non-code-specialized models show no such asymmetry: Code-LLaMA-7B exhibits a **reversed** pattern (Rust > Python at all layers), and Phi-3-mini shows no significant difference. See [RIGOR_EXPERIMENT.md](RIGOR_EXPERIMENT.md) for details.
150+
145151
### 3.2 Steering Successfully Boosts Attention
146152

147153
| Model | Baseline Rust | Target (Python) | Scale Needed | Achieved |
@@ -201,6 +207,8 @@ output = attn_weights @ V
201207
| `src/forward_qwen2.rs` | Post-softmax steering in Qwen2 attention |
202208
| `src/forward.rs` | Post-softmax steering in StarCoder2 attention |
203209
| `src/forward_gemma.rs` | Post-softmax steering in Gemma attention |
210+
| `src/forward_llama.rs` | Post-softmax steering in LLaMA/Code-LLaMA attention (v1.1.0) |
211+
| `src/forward_phi3.rs` | Post-softmax steering in Phi-3 attention (v1.1.0) |
204212
| `src/model.rs` | `forward_with_steering()`, `forward_steered_only()` |
205213
| `src/steering.rs` | Calibration utilities, `DOSE_LEVELS` |
206214
| `examples/steering_calibrate.rs` | Measure baseline attention |
@@ -244,14 +252,22 @@ cargo run --release --example steering_experiment -- \
244252
## 6. Next Steps
245253

246254
1. **Test on code generation tasks**: Measure if boosted attention improves test preservation in actual code generation
247-
2. **Compare with StarCoder2**: The paper notes StarCoder2 shows reversed patterns - steering may be counterproductive
248-
3. **Head-specific steering**: Identify which attention heads are most responsive to steering
249-
4. **Multi-layer steering**: Test steering across multiple layers simultaneously
250-
5. **Generation quality metrics**: Beyond KL divergence, measure BLEU/CodeBLEU on test-related tokens
255+
2. **Extend to other code-specialized models**: Run dose-response experiments on StarCoder2-3B and CodeGemma-7B, which both show the attention asymmetry (RIGOR_EXPERIMENT.md). StarCoder2's ablation results show extreme redundancy (ABLATION_RESULTS.md), so steering may behave differently.
256+
3. **Test on non-code-specialized models**: Run steering on Code-LLaMA-7B (reversed attention pattern) and Phi-3-mini (no differential). Since these models show no Python > Rust asymmetry, steering Rust attention upward may have fundamentally different effects — or none at all.
257+
4. **Head-specific steering**: Identify which attention heads are most responsive to steering
258+
5. **Multi-layer steering**: Test steering across multiple layers simultaneously
259+
6. **Generation quality metrics**: Beyond KL divergence, measure BLEU/CodeBLEU on test-related tokens
251260

252261
---
253262

254263
## 7. Raw Data Files
255264

256265
- Calibration output: `cargo run --release --example steering_calibrate -- --verbose > calibration_output.txt`
257266
- Experiment output: `cargo run --release --example steering_experiment -- --output results.json`
267+
268+
---
269+
270+
*Created: February 1, 2026*
271+
*Updated: February 9, 2026 (contextual updates for Code-LLaMA and Phi-3 attention findings)*
272+
*Steering scope: Dose-response experiments on Qwen-3B and Qwen-7B only. Steering infrastructure available for all 6 models via PlipBackend trait (v1.1.0).*
273+
*For: AIWare 2026 submission*

0 commit comments

Comments
 (0)