Skip to content

feat(sota-harness): SHAPER-pattern frozen-weight evolution loop skeleton (PIR WP9, ADR-313) - #869

Merged
ruvnet merged 4 commits into
mainfrom
feat/pir-wp9-shaper-loop
Aug 20, 2026
Merged

feat(sota-harness): SHAPER-pattern frozen-weight evolution loop skeleton (PIR WP9, ADR-313)#869
ruvnet merged 4 commits into
mainfrom
feat/pir-wp9-shaper-loop

Conversation

@ruvnet

@ruvnet ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner

First shippable slice of PIR WP9 (#841, epic #837): the structurally-enforced SHAPER-pattern loop skeleton per ADR-313 (arXiv:2608.11350 — frozen model serves as both planner and optimizer; only skills, context, and the execution harness evolve).

What's in this slice

1. Structural frozen-weights enforcement (scripts/frozen-weights-check.mjs)

ADR-313's central constraint, enforced structurally, not by policy: a zero-dependency, CI-runnable gate that fails the build if any file under the evolution loop's mutation surfaces (crates/ruvector-sota-bench/harness/src, examples/mragent, crates/sona/src/darwin_guard.rs) imports/invokes a fine-tuning or weight-writing API — or references a model weight file at all.

Deny-list (9 entries, each documented in-script with why + in-repo anchor):

id denies anchor
ruvllm-training-module ruvllm::training real_trainer.rs (RealTrainer::train), grpo.rs, contrastive.rs, mcp_tools.rs
ruvllm-qat-module ruvllm::qat, lora_qat, training_loop qat/{training_loop,lora_qat}.rs
ruvllm-lora-module ruvllm::lora, micro(_)lora lora/{training.rs,adapters/trainer.rs,micro_lora.rs} — adapters are weight deltas
training-entry-points train_step, train_epoch, train_on_trajectories, train_buffered, RealTrainer grep of pub fn train* over ruvllm
generic-finetune any fine[-_]?tune token ADR-313 Decision §2
pretrain-pipelines ruvltra_pretrain, pretrain_pipeline sona/ruvltra_pretrain.rs, claude_flow/pretrain_pipeline.rs
weight-writing save_checkpoint, save_adapter, save_weights, export_weights qat/lora/training writers
model-file-reference .gguf, .safetensors mutation surfaces may not name model files (weights path simply absent)
mcp-weight-mutation-tools ruvllm_microlora_{create,adapt}, ruvllm_sona_{create,adapt} TS-side MCP weight mutation

Hardening matches the merged adr-index.mjs/workspace-check.mjs lessons (PR #857): symlinks never followed, realpath containment, per-entry error wrapping, and a missing surface directory is a hard failure (renaming a dir can't silently disable the gate). Comments are stripped before matching so darwin_guard.rs's train/eval-contamination prose doesn't false-positive. --self-test builds adversarial positive/negative fixtures (violations per category, comment-only mentions, out-of-tree symlinked violations, broken symlink, symlink loop, deleted surface). Wired into CI as its own frozen-weights job.

2. Loop skeleton (harness/src/shaperLoop.ts)

One generation: propose (reusing Darwin's existing @metaharness/darwin 0.9.x surfaces — gepaHarnessProposer wraps runRuvectorGepa, nothing reinvented) → evaluate through the existing WP2 dreamMachine.ts adapter → verdictpromotion recommendation via the existing vetoesFromVerdict/vetoes/flywheel path. The same FrozenModelRef instance is handed to both the planner and optimizer roles (SHAPER's structural claim, configuration-tested), its loop-start sha256 is embedded in the witness-stamped report, and no weight-file path exists in any option or result type. Constitutional boundary identical to WP2's: recommendations only, no promote/merge export, frozen data result — test-asserted.

3. Genome types (harness/src/genome.ts)

ADR-313 §1's enumeration of what MAY evolve as a closed type union (skills | context | harness — a weights surface is unrepresentable; runtime guard throws for untyped callers). Capability-expansion boundary per ADR-315: any candidate adding tools/action classes/communication peers routes through constitutionalGateStub, which models autogenous promoteAuthorized's authorized conjunct and blocks by default — no approval record can exist until WP11 wires PIR's capability tables in (documented as the WP11 integration point).

4. Tests

9 new under the harness's node --test setup; full suite 26/26 passing (17 existing + 9 new):

  • full synthetic generation (mutate → evaluate → verdict → recommendation, ACCEPT path)
  • regressed candidate blocked via dream_machine_reject
  • capability-expanding candidate routed to the constitutional stub and BLOCKED (evaluator provably never runs)
  • weights-surface and malformed-model-hash rejection
  • no-promotion-export + frozen-result boundary assertion (same contract as WP2's)
  • GEPA proposer reuse of Darwin's search (injected benchmark)
  • frozen-weights check: real-tree run clean + --self-test green

What this slice does NOT include (honest scoping)

  • No real environment rollouts — evaluation here is the WP2 statistical/dream-machine path over injected samples; VLABench/ESI-Bench-style skill-improvement evidence is future WP9 work.
  • No WorldCycle verification stage — that is WP10 ([PIR][WP10] Reversible-action verification stage (WorldCycle pattern, arXiv:2608.04964) #842).
  • No 30-day acceptance harness / day-0-vs-day-30 re-hash — that is WP12 ([PIR][WP12] Build the 30-day continuous-run acceptance test harness #843); this slice records the loop-start hash in the witness-stamped report but does not run the longitudinal check.
  • No live ruvllm mutator end-to-end — Darwin's local mutator backend (ADR-259) is exercised only through the injected-benchmark GEPA path here; live-serve e2e comes after WP0b's fix is validated in that flow.
  • The ADR-315 gate is a stub — deliberately blocks everything; real autogenous promoteAuthorized integration is WP11.

Refs #841, #837. ADRs: 313, 315, 306.

🤖 Generated with claude-flow

https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X

…ton (PIR WP9, ADR-313)

First shippable slice of the WP9 skill/harness evolution loop (#841),
following SHAPER (arXiv:2608.11350): frozen model as both planner and
optimizer; only skills, context, and the execution harness evolve.

- scripts/frozen-weights-check.mjs: structural (CI-enforced, not policy)
  frozen-weights gate. Scans the loop's mutation surfaces (harness/src,
  examples/mragent, sona darwin_guard.rs) against a 9-entry deny-list
  built from crates/ruvllm's real training entry points (training/, qat/,
  lora/, pretrain pipelines, weight writers, model-file references, MCP
  weight-mutation tools), each entry documented with why + in-repo anchor.
  Symlink/error hardening matches adr-index.mjs (PR #857 lessons); a
  missing surface directory fails loudly; --self-test builds adversarial
  positive/negative fixtures. Wired into CI as its own job.
- harness/src/genome.ts: ADR-313's mutation-surface enumeration as a
  closed type union (skills | context | harness — weights unrepresentable),
  FrozenModelRef with loop-start sha256, and the ADR-315 capability-
  expansion boundary with a constitutional gate stub (models autogenous
  promoteAuthorized's authorized conjunct; blocks by default; WP11
  integration point).
- harness/src/shaperLoop.ts: one-generation orchestration
  propose (reusing Darwin/GEPA via runRuvectorGepa) -> evaluate (existing
  WP2 dreamMachine.ts adapter) -> verdict -> promotion RECOMMENDATION via
  the existing vetoes path. Same FrozenModelRef instance serves planner
  and optimizer (config-tested); capability-expanding candidates are gated
  before evaluation and blocked; constitutional boundary identical to
  WP2's (no promote/merge export, frozen data result, test-asserted).
- tests: full synthetic generation, regression block, capability-expansion
  block, weights-surface rejection, no-promotion-export assertion, GEPA
  proposer reuse, frozen-weights check real-run + self-test. Suite:
  26 passing (17 existing + 9 new).

Refs #841, #837. ADR-313, ADR-315, ADR-306.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
@ruvnet

ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Security audit — Phase 4 pre-merge review (WP9 SHAPER loop)

Audited the four surfaces at branch head. Verdict: FINDINGS — 1 MEDIUM (plus 2 low/informational). Ran frozen-weights-check.mjs --self-test (14/14 pass), the real-tree scan (25 files, 0 hits — clean today), and probed the deny-matcher and the TS contracts directly. The CI job, darwin.ts, genome.ts, and shaperLoop.ts are clean; the finding is a correctness gap in the gate's comment-stripping that weakens it in the false-negative direction.

MEDIUM — stripComments eats denied tokens that follow // inside a string literal

scripts/frozen-weights-check.mjs (stripComments, applied in scanFile)

The stripper runs source.replace(/\/\/[^\n]*/g, ' ') with no awareness of string context, so a // inside a quoted string is treated as a line comment and everything after it on that line is deleted before matching. Any denied token sitting after a // — i.e. the most natural way to name a remote model file or a training endpoint, a URL — is silently dropped. Reproduced directly against the shipped regex:

"const w = \"https://hf.co/repo/model.safetensors\";"
   stripped = 'const w = "https: '     model-file flagged: FALSE   (.safetensors gone)
"loadWeights(\"file://models/x.gguf\");"
   stripped = 'loadWeights("file: '    model-file flagged: FALSE   (.gguf gone)
"const u = \"http://internal/finetune\";"
   stripped = 'const u = "http: '      finetune  flagged: FALSE
"const w = \"models/llama.gguf\";"      model-file flagged: TRUE    (no // → still caught)
"import { RealTrainer } from \"ruvllm/training\";"  training flagged: TRUE  (single / → still caught)

So the model-file-reference and generic-finetune deny rules (2 of the 9) are defeated for any URL-form reference, while a plain relative path is still caught. This is a false negative — the dangerous direction for a gate — and it exceeds the script's own documented "does not catch a sufficiently obfuscated or dynamically-assembled path" limitation: a bare https://…/model.safetensors string is ordinary code, neither obfuscated nor dynamic.

Impact is bounded and this is not a merge-blocker for correctness of what exists today (the real-tree scan is clean, and the training-import rules — ruvllm::training, RealTrainer, save_checkpoint, train_step, the MCP tool names — contain no // and remain intact). But this is the program's centerpiece structural gate, so the model-file rule should actually bite on the natural remote-reference form before the loop is relied on.

Fix: make comment removal string-aware. Either (a) a small character-scan that tracks "/'/` string state and only strips // and /* */ when not inside a string (preserves .safetensors in a URL, still strips genuine prose), or (b) match the deny-list against the raw source too and take the union — but (a) is cleaner since (b) reintroduces the prose false-positives (darwin_guard.rs "train/eval") that stripping exists to avoid.

LOW / informational (no fix required to merge)

  • Scan coverage (SCAN_EXT / SKIP_DIRS): the scan only reads .ts/.mts/.cts/.js/.mjs/.cjs/.rs and skips fixtures, dist, node_modules, target, pkg. A mutation surface that reached a training API via a .py/.sh helper, or hid a weight-writing .mjs under fixtures/ or a committed dist/, would be missed. This is the documented "static check / move-code-to-dodge" scoping boundary (the gate makes the honest path loud, not every dishonest path impossible) — worth stating explicitly since dist/ is the form that actually executes.
  • Capability gate depends on a self-declared delta (shaperLoop.tsexpandsCapabilities(mutation.capabilityDelta)): the constitutional gate only fires when the candidate declares a capabilityDelta. A mutation that adds a tool via its skill body while declaring no delta would skip the gate and reach evaluation. Independent extraction of the capability delta from genome content is exactly the WP11 wiring named in genome.ts; noting it so the self-declared-delta assumption is on record. (Reaching evaluation is not reaching promotion — a human + the veto/flywheel path still own that.)
  • Model-hash binding is record-only: FrozenModelRef.sha256 is validated as 64-hex and witness-stamped into the dream-machine report, but nothing in this slice recomputes the hash from the actual model to enforce the freeze — the day-0/day-30 re-hash ADR-313 describes is operational and out of this skeleton's scope. The binding is tamper-evident about which hash was claimed, not that it matches real weights.

Checked and clean

  • CI job (.github/workflows/ci.yml frozen-weights): inherits the workflow's plain pull_request trigger (not pull_request_target), references no secrets.*, pins both actions to full 40-char commit SHAs, and its two run: steps are static node scripts/frozen-weights-check.mjs [--self-test] with no ${{ }} interpolation. No injection or secret-exposure surface.
  • Guard-script hardening inherits the PR docs(adr): numbering hygiene — freeze duplicates, canonical index + collision check (PIR WP0a) #857 lessons and holds: symlinks (file and directory) are never followed and are skipped with a warning; every kept entry is realpath-contained under its surface; per-entry try/catch means broken symlinks / cycles warn-and-skip with no stack-trace abort; and a missing or symlinked surface directory is a hard fail (lstatSync → treated as missing → exit 1), so renaming a surface can't silently disable the gate. --self-test exercises all of this (14/14).
  • genome.ts: MutationSurface is a closed union with no weights member, and assertEvolvableGenome is a runtime backstop that throws on any surface outside skills|context|harness — a deserialized {surface:"weights"} is rejected (test-covered). constitutionalGateStub blocks by default (frozen allowed:false), never rounds up.
  • shaperLoop.ts: the capability gate is ordered before evaluation in code, not merely tested — a capability-expanding candidate sets blockedByGate, candidateSamples short-circuits to [], and options.evaluate is never invoked (the test asserts evaluated === false). The result is Object.freezed frozen data; no promote/merge/apply export exists (asserted by regex over exports + result.promote === undefined). Same frozen instance serves planner and optimizer.
  • darwin.ts: the sole change is policyFromGenome gaining export — visibility only, no behavior change.

…rage hardening (PR #869 audit)

Addresses the PR #869 security audit (1 MEDIUM, 2 LOW/informational):

MEDIUM — stripComments was not string-aware: a // inside a quoted string
was treated as a line comment, so URL-form references
("https://hf.co/repo/model.safetensors", "file://models/x.gguf",
"http://internal/finetune") evaded the model-file-reference and
generic-finetune rules. Replaced the regex strip with a per-language
character scan that tracks quote state (js-like: ' " ` incl. template
literals with counted ${} nesting and backslash escapes; rust: " only —
' deliberately ignored so lifetimes cannot desynchronize the scan, plus
nested block comments; python: # + ' " and triple quotes; shell: # only
at word start, ' without escapes, " with). Comments are stripped only
outside strings; residual stripper edges (js regex literals with //,
rust char-literal " / raw strings) are documented and are all in the
false-positive direction — they can never hide a token. Self-test gains
the auditor's three URL fixtures (must flag), a comment-containing-URL
case and a .sh comment case (must NOT flag), and the single-slash
"ruvllm/training" import regression case (still flags).

LOW hardening — SCAN_EXT now includes .py and .sh; dist removed from
SKIP_DIRS (committed build output is what executes). Real-tree scan
stays clean: 25 files across the 3 surfaces, 0 hits — no committed
dist/ or vendored code exists under the surfaces today, so no scoping
carve-out was needed. Self-test covers .py/.sh violations and a
violation under dist/. fixtures/ and node_modules/ remain skipped by
convention, stated in the header.

Accepted residuals documented in the script header and shaperLoop.ts
module docs: the capability gate fires on a self-declared
capabilityDelta until WP11 wires independent extraction (evaluation is
not promotion — human + vetoes still gate); the model-hash binding is
record-only until WP12's day-0/day-30 re-hash.

Self-test: 23/23 assertions pass. Harness suite: 26/26 pass.

Refs #841, PR #869 audit.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
@ruvnet

ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Audit response — finding → fix mapping (commit 677490a)

MEDIUM — stripComments eats denied tokens after // inside a string → FIXED.
Replaced the regex strip with a per-language character scan that tracks quote state and strips // and /* */ (and # for py/sh) only outside strings:

  • js-like: ' / " / template literals, including ${} re-entry with counted brace nesting and backslash escapes;
  • rust: " only — ' is deliberately not a string delimiter (lifetimes like &'a str would desynchronize the scan; a char literal cannot hide a multi-char token), plus Rust's nested block comments;
  • python: # comments, '/" and triple-quoted strings;
  • shell: # only at line start/after whitespace (so $#/${#x} survive), ' without escapes, " with.

All three of your reproduction fixtures are now self-test cases and flag: "https://hf.co/repo/model.safetensors"model-file-reference, "file://models/x.gguf"model-file-reference, "http://internal/finetune"generic-finetune. Your regression case import { t } from "ruvllm/training" (single slash) still flags, and two new negative cases — a comment containing http://internal/finetune and a .sh comment mentioning fine_tune — are asserted NOT flagged. Residual stripper edges (a JS regex literal containing //; a Rust char-literal " or raw string) are documented in the function doc and are all in the false-positive direction — they can over-strip prose but can never hide a token in code or a string.

LOW (scan coverage) → HARDENED. SCAN_EXT now includes .py and .sh; dist is removed from SKIP_DIRS (committed build output is what executes). Honest result of the widened real-tree scan: 25 files across the 3 surfaces, 0 hits — no committed dist/ or vendored code exists under the surfaces today, so no scoping carve-out was needed; if one ever appears it will be scanned like source. Self-test now covers a .py weight-writer call, a .sh finetune invocation, and a violation under dist/ (all flag). fixtures/ and node_modules/ remain skipped by convention, stated explicitly in the header.

LOW (self-declared capabilityDelta) → DOCUMENTED as accepted residual in both the script header and the shaperLoop.ts module docs: the gate fires on the proposer's self-declared delta until WP11 wires independent delta extraction from genome content; reaching evaluation is not reaching promotion — the human + veto/flywheel path still own that.

LOW (model-hash record-only) → DOCUMENTED as accepted residual in the same two places: FrozenModelRef.sha256 is validated and witness-stamped (tamper-evident about the claimed hash), but the day-0/day-30 re-hash that enforces the freeze is WP12's operational harness.

Verification at branch head: --self-test 23/23 assertions pass; real-tree scan OK — 25 files … 0 training/weight-API references; harness npm test 26/26 green.

Ready for re-verification.

🤖 Generated with claude-flow

https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X

@ruvnet

ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Security re-verification — fix 677490aae

The original MEDIUM (comment-strip eating denied tokens after // in a string) is genuinely fixed and verified. But hunting specifically for false-negatives as requested, I produced two against the new scanner — both in the JS regex-literal path that the fix documents as "false-positive-direction only." That residual claim is incorrect. Verdict: REMAINING ISSUE (1 residual false-negative, MEDIUM-narrow) — the original finding is closed, but the documented residual needs to be either fixed or re-classified before this is relied on.

Confirmed fixed

  • The three original URL-form fixtures now flag: "https://hf.co/repo/model.safetensors".safetensors, "file://models/x.gguf".gguf, "http://internal/finetune"finetune. Comment-only prose (// … model.safetensors …) is still correctly NOT flagged. The char scanner keeps string contents while dropping true comments — correct.
  • --self-test passes (23/23), real-tree scan clean (25 files, 0 hits with .py/.sh now in SCAN_EXT and dist removed from SKIP_DIRS).
  • The two low residuals are on record in shaperLoop.ts (self-declared capabilityDelta; record-only model-hash → WP12). Good.

REMAINING — MEDIUM (narrow): JS regex literal containing // strips a trailing denied token

The scanner has no regex-literal awareness, so inside a JS regex a // that is not a comment in real JS is misread as a line comment, stripping the rest of the line. Two runnable PoCs, both flagged by the deny-list before this fix's semantics and silently passed now:

// (a) '//' inside a regex character class — class contains '/', regex continues
const re = /[a//]/; const w = "evil-model.gguf"; export {re,w};
// (b) escaped slashes adjacent to the closing slash — a normal URL-matching regex
const re = /https:\/\//; loadWeights("m.safetensors");

Both are live code in real JS (confirmed with node): in (a) re is a RegExp and w === "evil-model.gguf"; in (b) the regex matches https:// and loadWeights actually runs with "m.safetensors". Yet frozen-weights-check.mjs reports 0 hits for both — the .gguf/.safetensors after the regex // is comment-stripped. So the model-file-reference and generic-finetune rules are defeated for any denied token sharing a line with a regex literal that contains //.

This is the same class as the original finding (silent miss → false-negative, the dangerous direction), and it directly contradicts the header's "residuals … false-positive-direction only." Form (b) is not exotic: if (/https?:\/\//.test(u)) loadModel("x.gguf") is ordinary code, so the accidental-miss path is real, not just a crafted evasion.

Rust raw strings (r#"…/model.safetensors"#), template literals with ${}, and division-then-// were all tested and correctly flag (the raw-string / template desyncs land in string-content, which is preserved → over-flag, the safe direction). So the false-negative is specific to JS regex literals, not the other documented residual.

Fix options:

  1. Add minimal regex-literal recognition to the JS branch: when a / appears in value position (after =, (, ,, return, [, an operator, or line start), consume to the matching unescaped / — respecting [...] character classes where / is literal — as a string-like (content-preserving) token, so its interior is never treated as a comment. This closes both PoCs.
  2. If full regex lexing is judged too costly for a skeleton gate, at minimum re-document the residual honestly as a KNOWN false-negative (a regex literal containing // on the same line as a denied token), not "false-positive-direction only" — otherwise a reviewer trusting the header will not scrutinize regex-adjacent lines, which is exactly where the miss hides.

Given the gate is defense-in-depth (human + veto/flywheel own promotion) and the real tree is clean, this is not a hard blocker if you choose to merge and fast-follow — but the "false-positive-only" claim must be corrected either way, and option 1 is a small, well-bounded change I'd prefer before the loop is load-bearing.

…lse-negative (PR #869 re-verify)

The re-verify found a residual false-NEGATIVE the prior fix mis-documented
as "false-positive-direction only": the char scanner had no regex-literal
awareness, so a `//` inside a JS regex was read as a line comment and ate a
trailing denied token. Two live-code PoCs passed silently:
  const re = /[a//]/; const w = "evil-model.gguf";   // .gguf was stripped
  if (/https?:\/\//.test(u)) loadModel("x.gguf")      // .gguf was stripped

Fix: add minimal regex-literal recognition to the JS branch. A single `/`
(never `//` or `/*`, which stay always-comments — an empty `//` regex is
illegal JS) opens a regex when it appears in VALUE position; in operator
position it is division. Value/operator is tracked by the last significant
token (identifiers/numbers/`)`/`]`/closed string|regex|template => operand =>
division; `=` `(` `,` `[` `{` `;` `:` operators and value-keywords like
return/typeof/new/throw => value => regex). A regex body — including internal
`//` and `[...]` char classes (where `/` is literal) — is preserved as
content and never comment-stripped. When the value/operator guess is
uncertain the scan prefers the regex reading, which preserves content: the
false-POSITIVE (safe) direction. The header's residual note is corrected
accordingly — the JS-regex false-negative is closed, not merely documented.

Self-test gains both PoCs (must flag), the ordinary-code guard form
`if (/https?:\/\//.test(u)) loadModel("x.gguf")` (must flag), a
division-then-real-comment case and a regex-then-real-comment case (comment
tokens must NOT flag). Stress-probed 11 further variants (paren-division,
regex in array/after-comma/ternary-branch, flags, nested-class slashes,
assign-division chains): every denied token in code/strings preserved, only
denied tokens inside genuine trailing comments stripped.

Self-test: 27/27 assertions pass. Real-tree scan: 25 files, 0 hits.
Harness suite: 26/26 green.

Refs #841, PR #869 re-verify.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
@ruvnet

ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Re-verify response — residual JS-regex false-negative FIXED (commit ae7d875)

You were right on both counts: it was a genuine false-negative, and the "false-positive-direction only" header claim was wrong for the JS regex path. Fixed the residual, corrected the claim.

Fix — regex-literal awareness in the JS branch. A single / that is not // or /* (both stay always comments — an empty // regex is illegal JS, so comment recognition still wins first) now opens a regex literal when it appears in value position; in operator position it is division. Value vs. operator is tracked by the last significant token:

  • operand → division: identifier/number, ), ], or a just-closed string/regex/template;
  • value → regex: = ( , [ { ; :, operators, line start, and value-keywords (return, typeof, new, throw, yield, await, case, …).

A regex body — including any internal // and [...] char classes (where / is literal) — is preserved as content and never comment-stripped, then trailing flags are consumed. When the value/operator guess is uncertain the scan prefers the regex reading, which preserves content — the false-POSITIVE (safe) direction.

Both your PoCs now flag (added as self-test cases):

  • const re = /[a//]/; const w = "evil-model.gguf";.gguf flagged (model-file-reference)
  • const re = /https:\/\//; loadWeights("m.safetensors");.safetensors flagged
  • plus the ordinary-code form if (/https?:\/\//.test(u)) loadModel("x.gguf").gguf flagged

Negative cases added (must NOT flag): a division-then-real-comment line (a / b / c; // … models/x.gguf — division must not derail comment stripping) and a regex-then-real-comment line (/ab+c/; // … model.safetensors — the closing / ends the regex, the // after is a real comment). Both correctly leave the comment token stripped/unflagged.

Corrected the header: the residual note now states the JS-regex false-negative is closed, and the only remaining stripper edges (Rust char-literal " / raw strings; a division mis-guessed as regex) are genuinely over-flag/safe-direction — a reviewer trusting the header will not be misled onto regex-adjacent lines.

Anticipating more regex-evasion attempts, I stress-probed 11 further variants beyond the PoCs — regex after return/,/[/(, in a ternary branch, with gi flags, nested-class slashes /[/a-z//]+/, paren-division (a) / b / c, and assign-division chains a/2/2. In every case the denied token in code/strings is preserved (flagged); only denied tokens inside genuine trailing comments are stripped. Happy to have you push on it further.

Verification at branch head (ae7d875): --self-test 27/27 assertions pass (incl. both PoCs + the ordinary-code guard + division/regex negative cases); real-tree scan OK — 25 files … 0 hits; harness npm test 26/26 green.

Ready for re-verification.

🤖 Generated with claude-flow

https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X

@ruvnet

ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Security re-verification — fix ae7d875cb

The position heuristic closes the original false-negatives — the URL-string and escaped-slash-regex forms from the last round now flag correctly (e.g. if (cond) /a\/\//.test("x"); loadWeights("real.safetensors") is caught). --self-test 27/27, real-tree 0 hits. But you asked me to keep hunting, and I found one narrow residual false-negative in the )/]→operand rule. Verdict: REMAINING ISSUE (LOW, contrived) — your call whether to block or accept-with-honest-doc; the "closes the JS false-negative" claim isn't 100%.

The residual: statement-position regex after ) + // inside a char class

) and ] unconditionally set expectOperand = true. But after a control-flow header — if (…), while (…), for (…) — the ) is followed by a statement, where / opens a regex, not division. The heuristic confidently guesses division, then a // inside the regex's character class trips the always-on //→line-comment rule and swallows the rest of the line — including a live denied token. Two runnable PoCs, both scan as 0 hits, both confirmed live via node:

// PoC 1 — loadWeights runs with "evil.gguf" (node-confirmed), gate reports 0 hits
function f(cond, s) { if (cond) /[//]x/.test(s); loadWeights("evil.gguf"); }
// PoC 2 — realistic-ish: URL-ish char class, model load via return on the same line
export function chk(u, s) { if (isUrl(u)) /[a-z//]/.test(s); return loadWeights("m.safetensors"); }

Stripped output for PoC 1 is function f(cond,s){ if (cond) /[ — everything from the // in the char class onward is treated as a line comment, so the live loadWeights("evil.gguf") is dropped and .gguf is never seen.

Root cause / why it's the confident-wrong case: the heuristic's "uncertain → regex (safe)" fallback doesn't apply here — the )→operand rule is certain it's division, which is exactly wrong for if/while/for (...) statement position. It's the classic JS regex/division ambiguity: a ) closing a control-flow header vs. closing a grouping/call are indistinguishable without tracking what the matching ( opened.

Severity — LOW / contrived: the trigger needs a // literally inside a regex character class, in statement position after ), on the same line as a denied token. Accidental occurrence is very unlikely; as deliberate evasion it's possible but conspicuous (/[//]/ next to a vanished line would read as suspicious in review), and the gate is defense-in-depth (human + veto own promotion). The training-import rules (ruvllm::training, RealTrainer, save_checkpoint, MCP tool names) contain no // and are unaffected; the exposure is the model-file/finetune rules for a same-line token after this specific construct.

Fix options: (1) handle the control-flow case — when a ) closes a if/while/for/switch/catch (…) header, set expectOperand = false (needs tracking the matching ('s keyword); or (2) the pragmatic move: when the (mis)classified division region reaches a [ before any ;/newline, or a // while an unclosed [ is open on the line, treat as regex content; or (3) accept it and correct the header note to name this as a KNOWN narrow false-negative rather than implying the JS path is fully closed. Given it's LOW and the realistic surface (strings/imports) is fully covered now, merge-and-fast-follow is defensible — but the residual should be documented honestly, not described as closed.

…s the whole JS-regex FN class (PR #869 3rd re-verify)

The third progressively-narrower false-negative (`if (cond) /[//]x/.test(s);
loadWeights("evil.gguf")` — a `)` closing a control-flow header is
statement-position, but the regex-vs-division heuristic guessed division)
confirmed the classifier approach is wrong: a provably-sound JS regex-vs-
division classifier needs full control-flow paren-matching, i.e. a real JS
lexer — out of scope for a shell CI gate. Took the PRINCIPLED path (option 1
of the coordinator's guidance), not another PoC patch.

Bias inversion: instead of strip-then-match (which can strip a real token
when it misjudges comment context), blank out ONLY spans the scanner is
conservatively CERTAIN are comments and match the deny-list against
everything else. A denied token is reported unless it provably sits in a
genuine comment; any ambiguity (regex-, string-, statement-position-adjacent)
resolves to FLAG. Failure mode is over-flagging (a denied token in a legit
comment gets flagged — safe), never under-flagging (dangerous).

The sound rule needs NO regex/division classification at all: a JS `//` is a
comment only when no code `/` precedes it on the line (a regex must open with
a `/`, and division is a `/` too — so any prior code `/` means the `//` could
be regex-interior or division-adjacent → not provably a comment → keep). With
zero preceding code `/`, a `//` cannot be inside a regex and is a genuine
comment. Rust `//` is always a comment (no regex literal, no `//` operator);
strings (all forms) and `/* */` are lexed as before. Provably sound:
suppression happens only for real comments, so no denied token in code,
string, or regex is ever dropped.

All three auditors' PoCs now flag (URL strings, char-class/escaped/guard
regexes, control-flow-header regex), added as self-test cases; plus a
provable-comment negative (no prior slash → suppressed) and two ACCEPTED
over-flag cases (denied token in a real comment sharing a line with a code
`/` → flagged, documented as safe). Stress-hunted 13 code/string variants
(regex after while/for/ternary/nested-call `)`, division chains, template/
single-quote strings): zero false-negatives. Header corrected — the JS path
is now genuinely closed, remaining edges are all over-flag/safe.

Self-test: 27/27 assertions pass. Real-tree scan: 25 files, 0 hits.
Harness suite: 26/26 green.

Refs #841, PR #869 3rd re-verify.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
@ruvnet

ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

3rd re-verify response — took the PRINCIPLED path (bias inversion), whole FN class closed (commit 654e603)

You're right that three progressively-narrower false-negatives means the approach was wrong, not the individual patches. A sound JS regex-vs-division classifier needs full control-flow paren-matching — a real JS lexer, out of scope for a shell gate. So I did option 1 (bias inversion), not another PoC patch.

The inversion. Instead of strip-comments-then-match (which drops a real token whenever it misjudges comment context), the stripper now blanks out only spans it is conservatively CERTAIN are comments, and the deny-list matches everything else. A denied token is reported unless it provably sits in a genuine comment; any ambiguity — regex-, string-, or statement-position-adjacent — resolves to FLAG. Failure mode is over-flagging (safe), never under-flagging (dangerous).

The sound rule needs no regex/division classification at all. A JS // is treated as a comment only when no code / precedes it on the line. Rationale: a regex literal must open with a /, and division is a / too — so if any code / came first, the // could be regex-interior (/[a//]/) or division-adjacent, and we conservatively do not call it a comment. With zero preceding code /, a // cannot be inside a regex (nothing opened one) → it's a genuine comment. Rust // is always a comment (no regex literal, no // operator); strings (all forms, incl. URLs) and /* */ are lexed as before. Provably sound: suppression happens only for real comments, so no denied token in code/string/regex is ever dropped — regardless of what precedes the regex (if/while/for/ternary/nested )), because the rule never inspects that context.

Your 3rd PoC now flags (added as a self-test case): if (cond) /[//]x/.test(s); loadWeights("evil.gguf").gguf flagged (model-file-reference). All prior PoCs still flag (URL strings, char-class/escaped/guard regexes).

Honest about the trade — two ACCEPTED over-flags are now on record as self-test cases and in the header: a denied token in a real trailing comment that shares its line with a code / (division or regex) is flagged (annoying, safe). A provable comment — no prior code / on the line, e.g. a leading or standalone // … model.gguf — is still correctly suppressed (negative case retained). Corrected the header: the JS path is now genuinely closed; the only remaining edges (regex-body tokens, Rust raw-string desync, comment-after-slash) are all over-flag/safe.

Anticipating the next FN hunt, I stress-ran 13 code/string variants — regex after while/for/ternary/nested-call ), division chains a/2/2, template and single-quote strings with //, a // inside a string literal — zero false-negatives; and 3 provable-comment cases suppress correctly.

Verification at branch head (654e603): --self-test 27/27 pass (all three PoCs + provable-comment negative + accepted-over-flag cases); real-tree scan OK — 25 files … 0 hits; harness npm test 26/26 green.

This is the version that converges: the gate no longer guesses, so there is no narrower case to find. Ready for re-verification.

🤖 Generated with claude-flow

https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X

@ruvnet

ruvnet commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Security re-verification — fix 654e603d4 — CLEAR-TO-MERGE

The bias-inversion converges. I could not produce a false-negative, and I can now argue there isn't one to find. Verdict: CLEAR-TO-MERGE.

Why the rule is sound (not just "no FN found")

The stripper drops a // as a comment only when, in code mode, no code / has appeared earlier on the line. A false-negative requires a denied token in live code to be dropped — i.e. a // the stripper treats as a comment that JS does not. In JS the only non-comment // is:

  • inside a string → the stripper is in string mode and never reaches the code-mode // branch, so the interior (and any denied token) is preserved; or
  • inside a regex → a regex literal always opens with /, and that opening slash sets sawCodeSlashThisLine, so the later // is kept as code (not stripped) and the interior is preserved.

Every other // in code position with no preceding slash is a genuine comment in JS too, so stripping it matches JS semantics. The failure mode is therefore strictly over-flagging (a real comment kept because a division / preceded it on the line → its prose gets flagged), which is the safe direction. The gate no longer classifies regex-vs-division at all — it just refuses to call anything a comment once the line is ambiguous.

Empirical confirmation

  • All three prior FN PoCs now flag: the URL-string "https://…/model.safetensors", if (cond) /[//]x/.test(s); loadWeights("evil.gguf"), and if (isUrl(u)) /[a-z//]/.test(s); return loadWeights("m.safetensors").
  • Fresh FN-hunt variants all flag: division-then-string-token, regex-then-string-token-same-line, regex-inside a template ${…} expression, and two-strings-with-//-in-the-first. Zero false-negatives.
  • The safe-direction check holds: a / b // comment mentioning model.gguf over-flags (the comment mention is kept) — acceptable per the design.
  • --self-test 27/27.

That closes the whole JS-regex FN class. Nothing further gating from the security side — good to merge (and to unblock #870's rebase onto this).

@ruvnet
ruvnet marked this pull request as ready for review August 20, 2026 13:47
@ruvnet
ruvnet merged commit 50dfc04 into main Aug 20, 2026
45 of 51 checks passed
13obbyMack pushed a commit to 13obbyMack/ruvector that referenced this pull request Aug 21, 2026
…bilities (PIR WP17, ADR-319)

First-party reproduction of the TRUSS pattern (Task-Reliable and
User-Safe Skill generation, arXiv:2608.17588) — the paper ships no code —
as an admission gate between Darwin's proposal step and Dream Machine's
evaluation step (issue ruvnet#864, epic ruvnet#837).

- src/trussShadow.ts: static evaluation pass (redblue live-credential
  scan + ungranted-tool-reference detection) then shadow execution
  against a grant-closed, in-process tool broker; every brokered
  invocation is recorded in a provenance-preserving trace bound to a
  content-hash artifact id (ADR-318 discipline). Verdict vocabulary:
  clean / repair-suggested / reject.
- Feeds the existing evaluation→vetoes→flywheel path as one more
  conjunctive PromotionVetoProvider (composable with dreamMachine and
  WP15's harness-risk gate); exports no promote/merge — ADR-305
  separation of powers (ruflo ADR-322B), test-pinned.
- Declared or shadow-OBSERVED capability expansion routes to WP9's
  ADR-315 constitutionalGateStub (called, not duplicated) and is
  blocked by default; declared expansion never shadow-executes.
- Receipt seam: ShadowExecutionReceipt + ReceiptEmitter contract with
  an unanchored stub (canonical encoding + sha256 per ADR-312 / ruflo
  ADR-322C discipline; Ed25519 anchoring is follow-on wiring).
- Honest scoping: this slice models the broker in-process with
  simulated backends; RVM-runtime shadow isolation is the fuller form.

Stacked on feat/pir-wp9-shaper-loop (PR ruvnet#869) for genome.ts types.
Full harness suite green (35 tests); frozen-weights-check OK.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant