feat(sota-harness): SHAPER-pattern frozen-weight evolution loop skeleton (PIR WP9, ADR-313) - #869
Conversation
…ton (PIR WP9, ADR-313) First shippable slice of the WP9 skill/harness evolution loop (#841), following SHAPER (arXiv:2608.11350): frozen model as both planner and optimizer; only skills, context, and the execution harness evolve. - scripts/frozen-weights-check.mjs: structural (CI-enforced, not policy) frozen-weights gate. Scans the loop's mutation surfaces (harness/src, examples/mragent, sona darwin_guard.rs) against a 9-entry deny-list built from crates/ruvllm's real training entry points (training/, qat/, lora/, pretrain pipelines, weight writers, model-file references, MCP weight-mutation tools), each entry documented with why + in-repo anchor. Symlink/error hardening matches adr-index.mjs (PR #857 lessons); a missing surface directory fails loudly; --self-test builds adversarial positive/negative fixtures. Wired into CI as its own job. - harness/src/genome.ts: ADR-313's mutation-surface enumeration as a closed type union (skills | context | harness — weights unrepresentable), FrozenModelRef with loop-start sha256, and the ADR-315 capability- expansion boundary with a constitutional gate stub (models autogenous promoteAuthorized's authorized conjunct; blocks by default; WP11 integration point). - harness/src/shaperLoop.ts: one-generation orchestration propose (reusing Darwin/GEPA via runRuvectorGepa) -> evaluate (existing WP2 dreamMachine.ts adapter) -> verdict -> promotion RECOMMENDATION via the existing vetoes path. Same FrozenModelRef instance serves planner and optimizer (config-tested); capability-expanding candidates are gated before evaluation and blocked; constitutional boundary identical to WP2's (no promote/merge export, frozen data result, test-asserted). - tests: full synthetic generation, regression block, capability-expansion block, weights-surface rejection, no-promotion-export assertion, GEPA proposer reuse, frozen-weights check real-run + self-test. Suite: 26 passing (17 existing + 9 new). Refs #841, #837. ADR-313, ADR-315, ADR-306. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
Security audit — Phase 4 pre-merge review (WP9 SHAPER loop)Audited the four surfaces at branch head. Verdict: FINDINGS — 1 MEDIUM (plus 2 low/informational). Ran MEDIUM —
|
…rage hardening (PR #869 audit) Addresses the PR #869 security audit (1 MEDIUM, 2 LOW/informational): MEDIUM — stripComments was not string-aware: a // inside a quoted string was treated as a line comment, so URL-form references ("https://hf.co/repo/model.safetensors", "file://models/x.gguf", "http://internal/finetune") evaded the model-file-reference and generic-finetune rules. Replaced the regex strip with a per-language character scan that tracks quote state (js-like: ' " ` incl. template literals with counted ${} nesting and backslash escapes; rust: " only — ' deliberately ignored so lifetimes cannot desynchronize the scan, plus nested block comments; python: # + ' " and triple quotes; shell: # only at word start, ' without escapes, " with). Comments are stripped only outside strings; residual stripper edges (js regex literals with //, rust char-literal " / raw strings) are documented and are all in the false-positive direction — they can never hide a token. Self-test gains the auditor's three URL fixtures (must flag), a comment-containing-URL case and a .sh comment case (must NOT flag), and the single-slash "ruvllm/training" import regression case (still flags). LOW hardening — SCAN_EXT now includes .py and .sh; dist removed from SKIP_DIRS (committed build output is what executes). Real-tree scan stays clean: 25 files across the 3 surfaces, 0 hits — no committed dist/ or vendored code exists under the surfaces today, so no scoping carve-out was needed. Self-test covers .py/.sh violations and a violation under dist/. fixtures/ and node_modules/ remain skipped by convention, stated in the header. Accepted residuals documented in the script header and shaperLoop.ts module docs: the capability gate fires on a self-declared capabilityDelta until WP11 wires independent extraction (evaluation is not promotion — human + vetoes still gate); the model-hash binding is record-only until WP12's day-0/day-30 re-hash. Self-test: 23/23 assertions pass. Harness suite: 26/26 pass. Refs #841, PR #869 audit. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
Audit response — finding → fix mapping (commit 677490a)MEDIUM —
All three of your reproduction fixtures are now self-test cases and flag: LOW (scan coverage) → HARDENED. LOW (self-declared capabilityDelta) → DOCUMENTED as accepted residual in both the script header and the LOW (model-hash record-only) → DOCUMENTED as accepted residual in the same two places: Verification at branch head: Ready for re-verification. 🤖 Generated with claude-flow |
Security re-verification — fix
|
…lse-negative (PR #869 re-verify) The re-verify found a residual false-NEGATIVE the prior fix mis-documented as "false-positive-direction only": the char scanner had no regex-literal awareness, so a `//` inside a JS regex was read as a line comment and ate a trailing denied token. Two live-code PoCs passed silently: const re = /[a//]/; const w = "evil-model.gguf"; // .gguf was stripped if (/https?:\/\//.test(u)) loadModel("x.gguf") // .gguf was stripped Fix: add minimal regex-literal recognition to the JS branch. A single `/` (never `//` or `/*`, which stay always-comments — an empty `//` regex is illegal JS) opens a regex when it appears in VALUE position; in operator position it is division. Value/operator is tracked by the last significant token (identifiers/numbers/`)`/`]`/closed string|regex|template => operand => division; `=` `(` `,` `[` `{` `;` `:` operators and value-keywords like return/typeof/new/throw => value => regex). A regex body — including internal `//` and `[...]` char classes (where `/` is literal) — is preserved as content and never comment-stripped. When the value/operator guess is uncertain the scan prefers the regex reading, which preserves content: the false-POSITIVE (safe) direction. The header's residual note is corrected accordingly — the JS-regex false-negative is closed, not merely documented. Self-test gains both PoCs (must flag), the ordinary-code guard form `if (/https?:\/\//.test(u)) loadModel("x.gguf")` (must flag), a division-then-real-comment case and a regex-then-real-comment case (comment tokens must NOT flag). Stress-probed 11 further variants (paren-division, regex in array/after-comma/ternary-branch, flags, nested-class slashes, assign-division chains): every denied token in code/strings preserved, only denied tokens inside genuine trailing comments stripped. Self-test: 27/27 assertions pass. Real-tree scan: 25 files, 0 hits. Harness suite: 26/26 green. Refs #841, PR #869 re-verify. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
Re-verify response — residual JS-regex false-negative FIXED (commit ae7d875)You were right on both counts: it was a genuine false-negative, and the "false-positive-direction only" header claim was wrong for the JS regex path. Fixed the residual, corrected the claim. Fix — regex-literal awareness in the JS branch. A single
A regex body — including any internal Both your PoCs now flag (added as self-test cases):
Negative cases added (must NOT flag): a division-then-real-comment line ( Corrected the header: the residual note now states the JS-regex false-negative is closed, and the only remaining stripper edges (Rust char-literal Anticipating more regex-evasion attempts, I stress-probed 11 further variants beyond the PoCs — regex after Verification at branch head (ae7d875): Ready for re-verification. 🤖 Generated with claude-flow |
Security re-verification — fix
|
…s the whole JS-regex FN class (PR #869 3rd re-verify) The third progressively-narrower false-negative (`if (cond) /[//]x/.test(s); loadWeights("evil.gguf")` — a `)` closing a control-flow header is statement-position, but the regex-vs-division heuristic guessed division) confirmed the classifier approach is wrong: a provably-sound JS regex-vs- division classifier needs full control-flow paren-matching, i.e. a real JS lexer — out of scope for a shell CI gate. Took the PRINCIPLED path (option 1 of the coordinator's guidance), not another PoC patch. Bias inversion: instead of strip-then-match (which can strip a real token when it misjudges comment context), blank out ONLY spans the scanner is conservatively CERTAIN are comments and match the deny-list against everything else. A denied token is reported unless it provably sits in a genuine comment; any ambiguity (regex-, string-, statement-position-adjacent) resolves to FLAG. Failure mode is over-flagging (a denied token in a legit comment gets flagged — safe), never under-flagging (dangerous). The sound rule needs NO regex/division classification at all: a JS `//` is a comment only when no code `/` precedes it on the line (a regex must open with a `/`, and division is a `/` too — so any prior code `/` means the `//` could be regex-interior or division-adjacent → not provably a comment → keep). With zero preceding code `/`, a `//` cannot be inside a regex and is a genuine comment. Rust `//` is always a comment (no regex literal, no `//` operator); strings (all forms) and `/* */` are lexed as before. Provably sound: suppression happens only for real comments, so no denied token in code, string, or regex is ever dropped. All three auditors' PoCs now flag (URL strings, char-class/escaped/guard regexes, control-flow-header regex), added as self-test cases; plus a provable-comment negative (no prior slash → suppressed) and two ACCEPTED over-flag cases (denied token in a real comment sharing a line with a code `/` → flagged, documented as safe). Stress-hunted 13 code/string variants (regex after while/for/ternary/nested-call `)`, division chains, template/ single-quote strings): zero false-negatives. Header corrected — the JS path is now genuinely closed, remaining edges are all over-flag/safe. Self-test: 27/27 assertions pass. Real-tree scan: 25 files, 0 hits. Harness suite: 26/26 green. Refs #841, PR #869 3rd re-verify. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
3rd re-verify response — took the PRINCIPLED path (bias inversion), whole FN class closed (commit 654e603)You're right that three progressively-narrower false-negatives means the approach was wrong, not the individual patches. A sound JS regex-vs-division classifier needs full control-flow paren-matching — a real JS lexer, out of scope for a shell gate. So I did option 1 (bias inversion), not another PoC patch. The inversion. Instead of strip-comments-then-match (which drops a real token whenever it misjudges comment context), the stripper now blanks out only spans it is conservatively CERTAIN are comments, and the deny-list matches everything else. A denied token is reported unless it provably sits in a genuine comment; any ambiguity — regex-, string-, or statement-position-adjacent — resolves to FLAG. Failure mode is over-flagging (safe), never under-flagging (dangerous). The sound rule needs no regex/division classification at all. A JS Your 3rd PoC now flags (added as a self-test case): Honest about the trade — two ACCEPTED over-flags are now on record as self-test cases and in the header: a denied token in a real trailing comment that shares its line with a code Anticipating the next FN hunt, I stress-ran 13 code/string variants — regex after Verification at branch head (654e603): This is the version that converges: the gate no longer guesses, so there is no narrower case to find. Ready for re-verification. 🤖 Generated with claude-flow |
Security re-verification — fix
|
…bilities (PIR WP17, ADR-319) First-party reproduction of the TRUSS pattern (Task-Reliable and User-Safe Skill generation, arXiv:2608.17588) — the paper ships no code — as an admission gate between Darwin's proposal step and Dream Machine's evaluation step (issue ruvnet#864, epic ruvnet#837). - src/trussShadow.ts: static evaluation pass (redblue live-credential scan + ungranted-tool-reference detection) then shadow execution against a grant-closed, in-process tool broker; every brokered invocation is recorded in a provenance-preserving trace bound to a content-hash artifact id (ADR-318 discipline). Verdict vocabulary: clean / repair-suggested / reject. - Feeds the existing evaluation→vetoes→flywheel path as one more conjunctive PromotionVetoProvider (composable with dreamMachine and WP15's harness-risk gate); exports no promote/merge — ADR-305 separation of powers (ruflo ADR-322B), test-pinned. - Declared or shadow-OBSERVED capability expansion routes to WP9's ADR-315 constitutionalGateStub (called, not duplicated) and is blocked by default; declared expansion never shadow-executes. - Receipt seam: ShadowExecutionReceipt + ReceiptEmitter contract with an unanchored stub (canonical encoding + sha256 per ADR-312 / ruflo ADR-322C discipline; Ed25519 anchoring is follow-on wiring). - Honest scoping: this slice models the broker in-process with simulated backends; RVM-runtime shadow isolation is the fuller form. Stacked on feat/pir-wp9-shaper-loop (PR ruvnet#869) for genome.ts types. Full harness suite green (35 tests); frozen-weights-check OK. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
First shippable slice of PIR WP9 (#841, epic #837): the structurally-enforced SHAPER-pattern loop skeleton per ADR-313 (arXiv:2608.11350 — frozen model serves as both planner and optimizer; only skills, context, and the execution harness evolve).
What's in this slice
1. Structural frozen-weights enforcement (
scripts/frozen-weights-check.mjs)ADR-313's central constraint, enforced structurally, not by policy: a zero-dependency, CI-runnable gate that fails the build if any file under the evolution loop's mutation surfaces (
crates/ruvector-sota-bench/harness/src,examples/mragent,crates/sona/src/darwin_guard.rs) imports/invokes a fine-tuning or weight-writing API — or references a model weight file at all.Deny-list (9 entries, each documented in-script with why + in-repo anchor):
ruvllm-training-moduleruvllm::trainingreal_trainer.rs(RealTrainer::train),grpo.rs,contrastive.rs,mcp_tools.rsruvllm-qat-moduleruvllm::qat,lora_qat,training_loopqat/{training_loop,lora_qat}.rsruvllm-lora-moduleruvllm::lora,micro(_)loralora/{training.rs,adapters/trainer.rs,micro_lora.rs}— adapters are weight deltastraining-entry-pointstrain_step,train_epoch,train_on_trajectories,train_buffered,RealTrainerpub fn train*over ruvllmgeneric-finetunefine[-_]?tunetokenpretrain-pipelinesruvltra_pretrain,pretrain_pipelinesona/ruvltra_pretrain.rs,claude_flow/pretrain_pipeline.rsweight-writingsave_checkpoint,save_adapter,save_weights,export_weightsmodel-file-reference.gguf,.safetensorsmcp-weight-mutation-toolsruvllm_microlora_{create,adapt},ruvllm_sona_{create,adapt}Hardening matches the merged
adr-index.mjs/workspace-check.mjslessons (PR #857): symlinks never followed, realpath containment, per-entry error wrapping, and a missing surface directory is a hard failure (renaming a dir can't silently disable the gate). Comments are stripped before matching sodarwin_guard.rs's train/eval-contamination prose doesn't false-positive.--self-testbuilds adversarial positive/negative fixtures (violations per category, comment-only mentions, out-of-tree symlinked violations, broken symlink, symlink loop, deleted surface). Wired into CI as its ownfrozen-weightsjob.2. Loop skeleton (
harness/src/shaperLoop.ts)One generation: propose (reusing Darwin's existing
@metaharness/darwin0.9.x surfaces —gepaHarnessProposerwrapsrunRuvectorGepa, nothing reinvented) → evaluate through the existing WP2dreamMachine.tsadapter → verdict → promotion recommendation via the existingvetoesFromVerdict/vetoes/flywheel path. The sameFrozenModelRefinstance is handed to both the planner and optimizer roles (SHAPER's structural claim, configuration-tested), its loop-start sha256 is embedded in the witness-stamped report, and no weight-file path exists in any option or result type. Constitutional boundary identical to WP2's: recommendations only, no promote/merge export, frozen data result — test-asserted.3. Genome types (
harness/src/genome.ts)ADR-313 §1's enumeration of what MAY evolve as a closed type union (
skills | context | harness— a weights surface is unrepresentable; runtime guard throws for untyped callers). Capability-expansion boundary per ADR-315: any candidate adding tools/action classes/communication peers routes throughconstitutionalGateStub, which models autogenouspromoteAuthorized'sauthorizedconjunct and blocks by default — no approval record can exist until WP11 wires PIR's capability tables in (documented as the WP11 integration point).4. Tests
9 new under the harness's
node --testsetup; full suite 26/26 passing (17 existing + 9 new):dream_machine_reject--self-testgreenWhat this slice does NOT include (honest scoping)
promoteAuthorizedintegration is WP11.Refs #841, #837. ADRs: 313, 315, 306.
🤖 Generated with claude-flow
https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X