feat(agent-memory): TARL transactional memory ledger (PIR WP4, ADR-307)

[ruvnet]

Summary

Implements the PIR WP4 core slice (#840, depends-on/coordinates-with #839): the TARL (Transaction-Aware Reliable Ledgers) five-operation transactional memory ledger (arXiv:2608.03699) layered onto crates/ruvector-agent-memory, per ADR-307 (docs/adr/ADR-307-three-level-persistent-memory-livemem-tarl.md, branch feat/pir-adrs). WP3's three-tier architecture will sit on this ledger.

What's in the slice

• src/ops.rs — MemoryOp with the five TARL operations (add / ignore / revise-outdated-belief / reject-unreliable / defer-for-verification), LedgerState (Accepted | Pending | Rejected, each distinctly queryable), full-provenance TransitionRecord (op, prior state, new state, timestamp, reason, actor id, gate receipt), and the witness layer below.
• src/ledger.rs — TransactionalLedger applies every operation as an explicit state transition with full history retained. Each operation follows a stage → witness → commit discipline: all witness records for the operation (including containment cascades) are emitted before any state is applied, and a sink failure aborts the whole operation — no partial application is observable, and no mutation exists without a witness (ADR-134 INV-3 direction: an orphan witness without a mutation is possible on mid-emit failure; a mutation without a witness is not).
• Poisoning containment — each entry carries lightweight depends_on edges; a revise, reject, or defer on an Accepted entry transitively demotes every accepted dependent to Pending (each demotion is its own witnessed DependentDemotion transition), and accept refuses entries whose dependencies are not all Accepted. One bad update cannot silently keep downstream inferences accepted.

How acceptance routes through the proof gate

Honest scoping note: ruvector-agent-memory had no existing proof-gate usage (it wasn't even a workspace member — neither in members nor exclude, so cargo test -p ruvector-agent-memory could not run at all; this PR adds it to the workspace). Per the WP4 brief for that case, acceptance transitions (Pending → Accepted) are wired behind a ProofGate trait:

• The real implementation is feature-gated (proof-gate): WriteGateAdapter adapts ruvector-proof-gate's WriteGate (HashChainGate / MerkleGate) — the existing ADR-194/047 machinery (ADR-194-proof-gated-writes.md, ADR-047-proof-gated-mutation-protocol.md) — so every acceptance is admitted by the gate and its WriteReceipt (sequence + chain commitment) is retained in the transition's provenance and echoed into the witness record (capability_hash, aux). The gate is consulted before witnessing/committing; a denial mutates nothing.
• AlwaysAdmitGate exists for dev/tests and is documented as such; the adversarial test uses a deny-all gate to show denied writes never reach the accepted ledger.

Witness records (ADR-134 schema + ADR-322C alignment)

Every transition emits a LedgerWitnessRecord through a WitnessSink hook (no-op default; chain-verifying MemoryWitnessLog included). Fields, sizes, byte offsets, and FNV-1a prev_hash/record_hash chaining follow docs/adr/ADR-134-witness-schema-log-format.md; ledger transitions claim the unassigned 0xA0-0xA6 action-kind block. This slice deliberately does not depend on the rvm crates (RVM anchoring is WP8, cross-repo).

ADR-322C alignment (program's canonical witness/receipt contract, per ADR-312): the record is canonical-JSON-friendly (integer-only fields, stable serde field order) and carries an evidence_grade field using the three-value vocabulary recomputed / signature-verified / trusted-assertion, with the grade code bound into the hashed flags bits so it cannot drift from the chained layout. Follow-up: full record-shape conformance to the ruflo ADR-322C contract spec being extracted in ruflo#3066 (RFC 8785 JCS canonicalization, SHA-256 digests, Ed25519 domain-separated signing) is explicitly deferred to that spec landing.

Tests

18 existing tests (12 in ruvector-agent-memory, kept green) + 8 new integration tests (10 with --features proof-gate):

• five-op state-machine transitions incl. terminal-state guards
• poisoning containment: reject and revise on an accepted entry demote transitive dependents; unrelated accepted entries untouched; demoted dependents cannot be re-accepted while their premise is out
• adversarial deny-all gate: rejected/deferred/denied statements never reach the accepted ledger; a denied acceptance leaves no Accept record in history
• defer-for-verification round-trip (Pending → Accepted → Pending → Accepted, independent gate receipts)
• witness emission: exactly one record per committed transition; chain verifies from persisted records alone; injected tamper and deletion both detected
• property-ish replay: a 25-step mixed schedule's history replays to the exact final state map
• feature-gated: acceptance through a real HashChainGate with full cryptographic chain re-derivation

cargo test -p ruvector-agent-memory
  11 passed (lib) + 1 passed (bench bin) + 8 passed (tarl_ledger) — 0 failed
cargo test -p ruvector-agent-memory --features proof-gate
  11 passed + 1 passed + 10 passed — 0 failed
cargo clippy -p ruvector-agent-memory --features proof-gate --all-targets
  0 warnings (also fixed the 3 pre-existing compaction.rs lints that surfaced on joining the workspace)

References

• Issues: [PIR][WP3] Map LiveMem/TARL three-tier memory onto ruvllm context modules + ruvector-agent-memory #839 (WP3 three-tier memory), [PIR][WP4] Add TARL five-operation ledger (add/ignore/revise/reject/defer) onto ADR-194/047 proof-gated memory writes #840 (WP4 TARL ledger)
• TARL (Transaction-Aware Reliable Ledgers): arXiv:2608.03699
• ADR-307 (feat/pir-adrs), docs/adr/ADR-134-witness-schema-log-format.md, docs/adr/ADR-194-proof-gated-writes.md, docs/adr/ADR-047-proof-gated-mutation-protocol.md, ADR-252 (ADR-252-agent-memory-compaction.md)
• Evidence: docs/research/perpetual-intelligence-runtime/01-evidence-review.md item 4

🤖 Generated with claude-flow

[ruvnet]

Security audit — Phase 4 pre-merge review

Audited at 0f6341038. Every finding below was reproduced with a runnable PoC against this branch (throwaway worktree, no repo changes). The state machine, poisoning-containment cascade, and proof-gate wiring are sound — findings concentrate on the witness log's durability and tamper-evidence properties, not on the ledger logic.

Verdict: FINDINGS — 1 high, 4 medium, 2 low.

Checked and clean

• Proof-gate bypass routes: none found. Effect::SetState { state: Accepted } is constructed in exactly one place (accept(), src/ledger.rs:468); Effect::Insert always sets Pending. LedgerEntry has public fields but no &mut accessor escapes, and dependency-acceptance is enforced at src/ledger.rs:442-449.
• Dependency-cycle handling: no infinite loop or stack overflow possible. stage_demotions (src/ledger.rs:528) is an iterative BFS with a demoted visited-set, so it terminates even on a cycle — and cycles can't form anyway, since add() requires every depends_on id to already exist (src/ledger.rs:176-183) and ids are monotonically increasing, making the graph a DAG by construction.
• Test suite is genuinely thorough on state transitions, replay, and cascade ordering.

---

HIGH — a mid-operation witness-sink failure permanently breaks the persisted chain

src/ledger.rs:628-662 (emit_and_apply)

The emission loop persists records one at a time:

for s in &staged {
    self.sink.emit(&s.witness)?;   // early return on failure
}
self.last_witness_hash = prev;     // never reached on failure
self.witness_seq += staged.len() as u64;

Multi-record operations are the norm — reject_unreliable on an accepted entry with two dependents stages 3 records, revise_outdated_belief stages 2+N. If the sink fails on record k, records 0..k are already durable, but last_witness_hash and witness_seq are not advanced. Three consequences, all reproduced:

1. Orphan records. A witness record asserting Accepted -> Rejected for an entry that was never mutated is now in the durable log. This is the exact inverse of the crate's headline invariant — the doc guarantees "no witness, no mutation", but nothing guarantees "no mutation, no witness".
2. Sequence-number reuse. The next successful operation reuses the orphan's sequence.
3. Permanent chain fork. The next record chains off the pre-abort head, so verify_chain() returns false on the persisted log forever after — a single transient disk-full or fsync error destroys the auditability of the whole log.

PoC output (sink budget exhausted partway through reject_unreliable):

aborted op error: witness sink rejected record (mutation aborted): sink out of budget
persisted records after aborted op: 7 (was 6)
orphan: seq=6 action=0xA3 target=0 flags=0x3103
post-recovery add succeeded, id=3
next record after recovery: seq=6 action=0xA0     <-- duplicate sequence
verify_chain(persisted log) = false               <-- permanently broken

Exploit scenario: an attacker who can induce sink write failures (fill the disk, sever the remote-append connection) at a chosen moment both injects a false state-transition record and permanently invalidates chain verification, defeating any later audit — while the ledger keeps accepting writes and reporting success.

Suggested fix: make emission all-or-nothing. Either extend WitnessSink with a batch/transactional API (emit_batch(&[LedgerWitnessRecord]) that commits atomically), or add compensating rollback so a partial failure truncates the already-emitted prefix. If neither is feasible for a given sink, advance last_witness_hash/witness_seq past the aborted batch so subsequent records at least chain forward from the orphans and the log stays linearly verifiable.

---

MEDIUM — WitnessRejected is dead code; the headline invariant has zero test coverage

src/ops.rs:128-131, tests/tarl_ledger.rs

LedgerError::WitnessRejected is declared and given a Display impl but is never constructed anywhere in the crate, and both shipped sinks (NoopWitnessSink, MemoryWitnessLog) unconditionally return Ok(()). There is no FailingSink in the test suite, so the "no witness, no mutation" invariant — the crate's central safety claim, stated at src/ledger.rs:8-10 and src/ops.rs:317-319 — is never exercised. That is precisely why the HIGH above ships undetected.

Fix: add a failing-sink test that asserts (a) no state mutation, (b) no history append, and (c) no orphan record persisted / chain still verifies — including the multi-record cascade case.

---

MEDIUM — verify_chain() cannot detect tail truncation, and the doc says it can

src/ops.rs:341-355

The doc comment claims the walk "Detects tampering, reordering, and deletion". It detects middle deletion — which is what the test at tests/tarl_ledger.rs:280-287 exercises (records.remove(2) of 7). It cannot detect deletion of the newest records, because there is no committed head or expected length. Reproduced:

genuine log: 5 records, last action=0xA3 (the rejection)
after dropping the newest record: 4 records, verify=true  <-- rollback undetected

Exploit scenario: roll back the most recent reject_unreliable/DependentDemotion records so a poisoned entry stays Accepted on replay. verify_chain() reports the log as intact.

Fix: persist and check an expected record count / head commitment alongside the log, and add a tail-truncation case to the test.

---

MEDIUM — the chain is keyless, so verify_chain() gives no tamper evidence against a log writer

src/ops.rs:226-233, 286-315

This is faithful to ADR-134 §3 (FNV-1a chosen for speed, explicitly "not for cryptographic strength"), so it is not a spec violation — but the consequence is stronger than the code comments suggest, and this ledger is a higher-assurance consumer than ADR-134's baseline. Because no key is involved, an attacker with write access to the log recomputes every record_hash and relinks every prev_hash in one O(n) pass:

genuine log: 5 records, verify=true
after naive deletion of record 2: verify=false
after repair: 4 records, verify=true  <-- rejection erased, chain 'valid'

Worth noting for the roadmap: FNV-1a means anchoring the head does not rescue this either. FNV-1a's round function is multiplication by an odd constant mod 2^64, hence invertible, so a meet-in-the-middle second-preimage against a published head costs roughly 2^32 work with 8 controllable bytes — trivial, versus 2^128 for a cryptographic hash. If WP8 cross-repo anchoring is meant to provide real tamper evidence, the chain function has to change with it.

Fix: no change needed for this slice, but (a) soften the verify_chain doc comment — it detects accidental corruption and naive edits, not an adversary; and (b) wire ADR-134's WitnessSigner escape hatch (which ADR-134 designates for exactly this case) before the ledger becomes load-bearing for acceptance decisions.

---

MEDIUM — aux on the newest record is unauthenticated

src/ops.rs:305-314

compute_record_hash() covers bytes [0..48], which excludes aux at [56..64]; only chain_hash() (full 64 bytes) covers it, and that is checked solely via the next record's prev_hash. So for the most recent record, aux can be changed to anything and verify_chain() still passes:

mutated last record aux: 0 -> 16045690984833335023
verify after aux tamper = true

On an Accept record aux holds the first 8 bytes of the proof-gate commitment (src/ledger.rs:454) — i.e. the gate evidence itself is the unauthenticated field. The layout follows ADR-134, so the fix is a verifier change, not a layout change.

Fix: have verify_chain() require a terminating commitment (see the tail-truncation finding) so the last record's full 64 bytes are always covered by something.

---

LOW — evidence_grade can diverge from the flags bits it is supposedly bound to

src/ops.rs:269-271

The field doc states its code "is also bound into the hashed flags bits so it cannot drift from the signed layout". Nothing enforces that: verify_chain() never cross-checks the serde field against flags bits 12-15. A persisted record can advertise a stronger grade than the hash preimage records, and a consumer reading the named enum (the natural choice over bit-twiddling) gets the forged value:

tampered record 0 JSON: {..."flags":12290,..."evidence_grade":"signature-verified"}
verify after grade upgrade = true

Low because SignatureVerified is unused in this slice — but it is the grade that will matter at WP8.

Fix: add (flags >> 12) & 0xF == evidence_grade.code() to verify_chain(), or derive the field from flags on deserialize rather than storing it twice.

---

LOW — the proof gate advances even when the acceptance is never witnessed or applied

src/ledger.rs:453-475

gate.admit() is called before staging. If the witness sink then fails, the gate has already consumed a sequence and advanced its chain (HashChainGate mutates on admit), while nothing is witnessed and nothing is applied:

accept attempt 0..2: witness sink rejected record (mutation aborted)
gate sequence after 3 failed accepts = 3

The gate's chain therefore contains entries with no corresponding accepted entry, so gate sequence numbers can't be reconciled against the ledger during an audit. The comment at src/ledger.rs:451-452 correctly covers the denied case but not the granted-then-unwitnessed case.

Fix: note the asymmetry in the doc comment, and treat gate-vs-ledger sequence gaps as expected during reconciliation (or move the gate call after successful emission, if the gate's semantics allow).

---

Scanners: cargo audit on this branch reports RUSTSEC-2026-0258 (h2) and RUSTSEC-2026-0253 (lru, unsoundness). Both are pre-existing repo-wide — this PR's Cargo.lock delta adds only the ruvector-agent-memory entry (rand, serde, serde_json, ruvector-proof-gate), no new third-party dependencies.

[ruvnet]

Security findings addressed — 74193f5fd

All 7 findings from the Phase-4 audit are fixed or explicitly documented on this branch. Finding → fix:

HIGH — non-atomic witness emission (ledger.rs emit_and_apply)

Design chosen: atomic batch contract on the sink (WitnessSink::emit_batch), not compensating truncation in the ledger. emit_batch(&[LedgerWitnessRecord]) is now the required trait method with a documented all-or-nothing contract: on Ok every record is durably retained, on Err the sink must retain none (staging via temp-file/rename, transactional append, or internal truncation of a partial prefix — whichever fits the medium). emit() survives as a provided convenience delegating to it.

Why this design: only the sink knows how to make its medium atomic — the ledger cannot truncate a remote append log or un-fsync a file, so a ledger-side compensating rollback would just be a second fallible write path. With the contract in the sink, emit_and_apply emits each operation's chained records as one batch and advances last_witness_hash/witness_seq only after that single call succeeds, so both sides stay consistent by construction: no orphan records, no sequence reuse, no chain fork. Minimal API delta (one trait method; the crate is new in this PR, so no external implementors break), and both shipped sinks satisfy the contract trivially.

MED — invariant had zero test coverage

Added FailingSink (budget-based, honors the atomic contract, constructs WitnessRejected — no longer dead code) with two tests:

• failing_sink_single_op_no_witness_no_mutation — refused add: no state, no history, no orphan, chain verifies.
• failing_sink_mid_cascade_leaves_ledger_and_chain_consistent — the audit's exact scenario: reject_unreliable on an accepted base with 2 accepted dependents (3-record batch), budget models the medium failing at record 3. Asserts states/history/persisted log/head commitment all unchanged and verifying, then recovery: retry succeeds, sequences stay dense (r.sequence == i over the whole log), chain never forks, replay_history still matches.

MED — tail truncation undetectable

MemoryWitnessLog now maintains a head commitment — committed_count + committed_head (full-64-byte chain hash of the newest record), advanced atomically with each batch and exported via head_commitment() for out-of-band anchoring. verify_chain() requires the walk to terminate exactly at that commitment. Test tail_truncation_detected_by_head_commitment covers dropping the newest record and the whole newest operation (the rejection + demotion rollback scenario). Doc no longer overclaims.

MED — keyless FNV relinking

No crypto implemented here (WP8/ADR-312 own real signing), as prescribed. Module docs (ops.rs header, verify_chain, ledger.rs header) now state the chain is tamper-evident against accidental corruption and naive edits only, not against a log-writing adversary (including the ~2^32 second-preimage note against an anchored head), and name ADR-134 §9 WitnessSigner wiring as an explicit follow-up gate that MUST land before WP8 anchoring makes this log load-bearing.

MED — aux bytes unauthenticated

Fixed via the supplementary terminating commitment, not a schema change: the ADR-134-prescribed 48-byte record_hash preimage is untouched (layout stays ADR-134-compatible), but committed_head is the full-64-byte chain_hash() of the newest record, so the last record's aux (the proof-gate commitment on Accept) and record_hash are now always covered by something. Test aux_tamper_on_newest_record_detected flips aux on a trailing Accept record and asserts verification fails.

LOW — evidence_grade/flags cross-check

verify_chain() now rejects any record where (flags >> 12) & 0xF != evidence_grade.code(). Test evidence_grade_flags_divergence_detected forges a SignatureVerified upgrade on the serde field and asserts failure. (Chose the verifier-side check over a custom Deserialize so the persisted-log audit path catches it regardless of how records were loaded.)

LOW — gate.admit() asymmetry

Documented in accept(): the gate must run before staging because the witness record binds the receipt (aux = commitment prefix, capability_hash = its digest), and WriteGate::admit offers no rollback — so on a refused batch the gate sequence runs ahead. The comment specifies reconciliation semantics: gate admissions with no matching Accept record are expected benign gaps; the inverse (an Accept record with no gate admission) is the bypass signal. Moving the gate after emission was not viable without breaking the receipt binding.

Test evidence

cargo test -p ruvector-agent-memory
  11 unit + 1 bench-acceptance + 13 integration — all ok
cargo test -p ruvector-agent-memory --features proof-gate
  running 15 tests ... test result: ok. 15 passed; 0 failed
  (incl. failing_sink_mid_cascade_leaves_ledger_and_chain_consistent,
   tail_truncation_detected_by_head_commitment,
   aux_tamper_on_newest_record_detected,
   evidence_grade_flags_divergence_detected,
   failing_sink_single_op_no_witness_no_mutation)
cargo clippy -p ruvector-agent-memory --all-targets [--features proof-gate] — no lints
cargo fmt --check — clean

All 29 pre-existing tests stay green (two chain-tamper tests updated only to use the new Clone on MemoryWitnessLog instead of struct literals).

Deferred (by design): real cryptographic signing (ADR-134 WitnessSigner) — owned by WP8/ADR-312, now documented as a hard gate before anchoring. Out-of-band anchoring of head_commitment() is exposed but wiring it cross-repo is WP8 scope.

🤖 Generated with claude-flow

https://claude.ai/code/session_012Jib2gQyJpqCoo2xYAbb4X

[ruvnet]

Security re-verification — fix 74193f5fd

Re-audited the TARL ledger at the fix commit. Verdict: CLEAR-TO-MERGE. All seven findings (1 HIGH, 4 MED, 2 LOW) are closed. Ran both feature-set suites: default 13/13, --features proof-gate 15/15 — including the four new adversarial tests. Read the diff and probed the design against the three questions raised in the re-verify request.

Each finding, confirmed closed

• HIGH atomic emission: emit_and_apply now collects the operation's records and emits them via a single sink.emit_batch(&witnesses)?; the chain cursors (last_witness_hash, witness_seq) advance only after that returns Ok. WitnessSink::emit_batch carries a documented all-or-nothing contract. A refused batch can no longer orphan records, reuse a sequence, or fork the chain — proven by failing_sink_single_op_no_witness_no_mutation and the mid-cascade test below.
• MED invariant coverage: FailingSink (budget-limited, honors the atomicity contract) now exercises LedgerError::WitnessRejected. failing_sink_mid_cascade_leaves_ledger_and_chain_consistent fails the 3-record reject cascade at record 3 (budget 2) and asserts the dependents stay in their original state (base/d1/d2 all still Accepted), no history append, no orphan record, head commitment unchanged, chain still verifies — then recovery commits with dense sequences (sequence == i) and no fork.
• MED tail truncation + MED aux: MemoryWitnessLog maintains a head commitment (committed_count, committed_head=full-64-byte chain_hash of newest), and verify_chain now requires the walk to terminate exactly at it. Because chain_hash covers all 64 bytes, tampering the newest record's aux changes committed_head → detected (aux_tamper_on_newest_record_detected); dropping the newest record(s) makes count/head mismatch → detected (tail_truncation_detected_by_head_commitment).
• MED keyless FNV: module doc and verify_chain doc now state plainly that the chain is tamper-evident against accidental corruption / naive edits only (an adversary relinks it in one O(n) pass; FNV-1a is invertible so even an anchored head is ~2^32 second-preimageable), and name the ADR-134 WitnessSigner wiring as a required gate before WP8 anchoring.
• LOW evidence_grade drift: verify_chain cross-checks (flags>>12)&0xF == evidence_grade.code() (evidence_grade_flags_divergence_detected).
• LOW gate asymmetry: accept() documents the granted-then-unwitnessed case and instructs reconciliation to treat gate-ahead-of-ledger as a benign gap (a bypass would be the opposite — an Accept record with no gate admission).

Adversarial checks

• Can a sink violate its contract undetectably? For the two in-repo sinks, no: NoopWitnessSink stores nothing, and MemoryWitnessLog::emit_batch does an infallible extend_from_slice then advances the commitment — there is no partial-retain-then-error path in either, so both satisfy the contract by construction. For a future disk/remote sink the contract is documentation, not enforcement — which is the correct place to put the obligation at a trait boundary, and the doc spells out the required implementation (temp-file+rename / transactional append / compensating truncation). Acceptable.
• Is the head commitment beyond a log-writer's reach? No — it lives in the same MemoryWitnessLog struct as the records, so an adversary who can rewrite the records vec can also rewrite committed_count/committed_head. This is an honest residual, and it's acceptable: it's the same keyless-FNV trust boundary, the commitment closes tail-truncation/aux against accidental corruption and naive record-only edits, the doc says in-band verification cannot detect wholesale replacement, and head_commitment() is exported specifically to anchor the pair out-of-band. Real adversarial tamper-evidence is gated on the named WitnessSigner follow-up before WP8.
• Mid-cascade failure leaves dependents in original state? Yes — asserted directly (all three entries remain Accepted after the failed cascade).

Solid, faithful-to-ADR-134 hardening. Ship it — with the WitnessSigner wiring tracked as the hard gate before WP8 cross-repo anchoring makes this log load-bearing.