docs: record runbook multi-agent review

Author: dknauss

CHANGELOG.md

@@ -2,6 +2,9 @@
 All notable changes to this project will be documented in this file.
 
 Unreleased
+- Editorial review: Executed the first focused Runbook-only review pass with the single-model, multi-agent method and recorded phase outputs plus a prioritized synthesis in `wp-security-doc-review/rounds/2026-03-15/`.
+- Housekeeping: Closed out STATE.md to reflect Phase 2 completion, refreshed accumulated context, and cleared stale pending todos.
+- Backlog: Added three tracking items — VIP step-up auth citation for canonical docs, NordVPN cookie study monitoring, and IBM CODB shadow AI figure monitoring.
 - Editorial review: Bootstrapped a focused runbook-only review round in `wp-security-doc-review/rounds/2026-03-15/`, narrowing the next audit to the Operations Runbook while retaining the other three canonical docs as supporting references.
 - Release readiness: Added `wp-security-doc-review/wordpress-7.0-readiness-2026-03-15.md`, capturing the current WordPress 7.0 release target, current official PHP guidance, the canonical version-reference surface, and a cautious PHP 8.4 recommendation path.
 - Editorial review process hardening: Added a source-driven WP-CLI command-path validator backed by live `wp help` lookups, and upgraded the glossary coverage check so it keys off actual repeated usage across the canonical docs rather than requiring every watched term unconditionally.

wp-security-doc-review/REVISION-LOG.md

@@ -19,6 +19,7 @@ Chronological record of editorial rounds, changes applied, and commits. Each rou
 - The focused round keeps `WP-Operations-Runbook.md` as the primary review target and uses the Benchmark, Hardening Guide, and Style Guide as supporting references.
 - The WordPress 7.0 readiness brief is grounded in current official project references: WordPress 7.0 remains scheduled for April 9, 2026; WordPress.org currently recommends PHP 8.3+; and core presently describes PHP 8.4 as beta compatible rather than fully supported.
 - `bash tools/ci/review_preflight.sh` passed before model execution, so the round is mechanically clear to proceed.
+- The first review pass was executed with the single-model, multi-agent method. Phase outputs were recorded in `rounds/2026-03-15/phase1-runbook.md`, `phase1-benchmark-alignment.md`, `phase1-hardening-guide.md`, and `phase1-style-guide.md`, then merged into `rounds/2026-03-15/synthesis.md`.
 
 ---
 

wp-security-doc-review/rounds/2026-03-15/EXECUTION-GUIDE.md

@@ -8,7 +8,7 @@
 4. If the preflight finds a mechanical issue, either fix it first or record it explicitly before asking models to review the documents.
 5. Keep this round scoped to the Operations Runbook as the primary review target. Use the other three canonical docs as references when cross-document alignment matters.
 
-## Files To Upload
+## Files To Review
 
 1. `/wordpress-runbook-template/WP-Operations-Runbook.md`
 2. `/wp-security-benchmark/WordPress-Security-Benchmark.md`
@@ -28,16 +28,17 @@ Ask reviewers to prioritize:
 
 ## Output Files
 
-Save each model's output as:
-- `gemini-review.md`
-- `gpt-review.md`
-- `claude-review.md`
+Save each phase output as:
+- `phase1-runbook.md`
+- `phase1-benchmark-alignment.md`
+- `phase1-hardening-guide.md`
+- `phase1-style-guide.md`
 
 Place them in: `wp-security-doc-review/rounds/2026-03-15/`
 
 ## After Completion
 
-1. Save each model output in the round directory.
-2. Run synthesis and verify every merged finding ends in one archival state: `applied`, `rejected`, or `stale`.
+1. Save each phase output in the round directory.
+2. Run synthesis and verify every merged finding ends in one archival state: `applied`, `rejected`, or `stale` once the round closes.
 3. Update `README.md` to reflect the current round state.
 4. Re-verify cross-repo metrics after approved fixes land in the canonical repos.

wp-security-doc-review/rounds/2026-03-15/README.md

@@ -2,7 +2,7 @@
 
 ## Status
 
-Round bootstrapped for a runbook-only review. `bash tools/ci/review_preflight.sh` passed on 2026-03-15, so the round is mechanically clear for model execution.
+Round bootstrapped for a runbook-only review. `bash tools/ci/review_preflight.sh` passed on 2026-03-15, and the first single-model, multi-agent review pass is now complete.
 
 ## Primary Document Under Review
 
@@ -28,11 +28,26 @@ Round bootstrapped for a runbook-only review. `bash tools/ci/review_preflight.sh
 
 See: `review-prompt.md`
 
+## Review Method
+
+This round is using the single-model, multi-agent method documented in [single-model-multi-agent.md](/Users/danknauss/Documents/GitHub/ai-assisted-docs/wp-security-doc-review/methodology/single-model-multi-agent.md), adapted here for a focused runbook audit:
+
+1. deterministic preflight
+2. specialized parallel review agents
+3. local cross-agent synthesis in `synthesis.md`
+
+Phase 1 outputs recorded for this round:
+
+- `phase1-runbook.md`
+- `phase1-benchmark-alignment.md`
+- `phase1-hardening-guide.md`
+- `phase1-style-guide.md`
+
 ## Execution
 
 1. Run `bash tools/ci/review_preflight.sh`.
-2. Submit the Runbook plus the supporting references and `review-prompt.md` to each independent model.
-3. Save outputs as `gemini-review.md`, `gpt-review.md`, and `claude-review.md`.
+2. Run specialized review agents against the Runbook and supporting references.
+3. Record phase outputs in the round directory.
 4. Merge findings into `synthesis.md`.
 5. Close the round only after every merged finding ends as `applied`, `rejected`, or `stale`.
 
@@ -46,4 +61,4 @@ See: `review-prompt.md`
 
 ## Next Step
 
-Collect the independent review outputs, then synthesize the findings into `synthesis.md`.
+Review the synthesized findings in `synthesis.md`, decide what to apply in the canonical Runbook repo, and then update the disposition ledger as fixes land.

wp-security-doc-review/rounds/2026-03-15/phase1-benchmark-alignment.md

@@ -0,0 +1,10 @@
+# Phase 1 — Benchmark Alignment Findings
+
+Focused cross-document alignment pass comparing the runbook to [WordPress-Security-Benchmark.md](/Users/danknauss/Documents/GitHub/wp-security-benchmark/WordPress-Security-Benchmark.md).
+
+| Severity | File:line | Finding | Recommendation | Verification |
+|---|---|---|---|---|
+| High | `WP-Operations-Runbook.md:1368-1374` | Backup retention drifts from Benchmark 10.1. The runbook keeps weekly full-site snapshots for 8 weeks, while the Benchmark requires at least 90 days of weekly backups. | Raise weekly retention to at least 90 days and make daily/weekly retention policy explicit for both database and file backups. | Compare the runbook retention table to Benchmark 10.1 remediation at `WordPress-Security-Benchmark.md:1924-1928`. |
+| High | `WP-Operations-Runbook.md:1416-1419` | Backup storage expectations drift from Benchmark 10.1. The runbook's example syncs to generic remote storage without requiring encrypted, offsite storage inaccessible from the production web server. | Require encrypted offsite backup storage and document isolation from routine production-host access. | Compare the runbook backup workflow to Benchmark 10.1 at `WordPress-Security-Benchmark.md:1906-1908` and `WordPress-Security-Benchmark.md:1924-1928`. |
+| Medium | `WP-Operations-Runbook.md:628-665` | The runbook partially addresses public user enumeration via REST but omits the author-archive side (`/?author=N`) that Benchmark 5.4 treats as part of the same operational control. | Add author-archive mitigation and verification steps, or explicitly point to the section that implements it. | Run both checks from Benchmark 5.4: REST users endpoint and `?author=1`. |
+| Medium | `WP-Operations-Runbook.md:667-708` | The runbook operationalizes Benchmark 5.1 (2FA) but does not operationalize Benchmark 5.5 reauthentication for privileged actions. | Add a dedicated privileged-action reauthentication procedure with gated actions, verification, break-glass handling, and rollback. | Compare the runbook authentication section to Benchmark 5.5 at `WordPress-Security-Benchmark.md:1212-1235`. |

wp-security-doc-review/rounds/2026-03-15/phase1-hardening-guide.md

@@ -0,0 +1,12 @@
+# Phase 1 — Hardening Guide Alignment Findings
+
+Focused cross-document alignment pass comparing the runbook to [WordPress-Security-Hardening-Guide.md](/Users/danknauss/Documents/GitHub/wp-security-hardening-guide/WordPress-Security-Hardening-Guide.md).
+
+| Severity | File:line | Finding | Recommendation | Verification |
+|---|---|---|---|---|
+| Medium | `WP-Operations-Runbook.md:126-127`, `WP-Operations-Runbook.md:1019` | The runbook still presents `WordPress 6.4+` and `PHP 8.2+` as baseline environment values while the current document set has already moved toward newer version guidance. | Update the runbook to current version guidance or relabel these values as compatibility minima rather than recommended targets. | Compare against the Hardening Guide's current version framing and the WordPress 7.0 readiness brief. |
+| High | `WP-Operations-Runbook.md:669-687` | The runbook scopes required 2FA to administrator-capable accounts, while the Hardening Guide requires MFA/2FA for administrator and editor accounts and encourages broader enrollment. | Expand enforcement scope to include editor accounts at minimum and note optional broader rollout. | Compare against `WordPress-Security-Hardening-Guide.md:288-300`. |
+| Medium | `WP-Operations-Runbook.md:698-708` | The break-glass 2FA path disables the `two-factor` plugin globally, creating a site-wide MFA bypass window that conflicts with the Hardening Guide's stronger reauthentication posture. | Replace global plugin deactivation as the default recovery path with a per-user recovery workflow where possible, or require explicit approval, short duration, and compensating controls. | Compare against `WordPress-Security-Hardening-Guide.md:304-315`. |
+| High | `WP-Operations-Runbook.md:1545-1546` | The deployment procedure runs `wp plugin update --all` and `wp theme update --all` during production release deployment, which conflicts with the Hardening Guide's version-controlled, auditable deployment pipeline guidance. | Remove blanket production extension updates from deployment procedures and keep patching in dedicated update workflows. | Compare against the Hardening Guide's deployment-pipeline guidance in the software and operations sections. |
+| Medium | `WP-Operations-Runbook.md:129` | The runbook is framed as suitable for `Self-hosted/Managed/VPS`, but most procedures assume self-managed Linux/Nginx root access. | Narrow the scope to self-managed environments or add managed-hosting/provider-escalation branches. | Compare the environment framing in the runbook to the shared-responsibility language in the Hardening Guide. |
+| Medium | `WP-Operations-Runbook.md:2190-2199`, `WP-Operations-Runbook.md:2559-2560` | The compromise procedure resets administrator passwords and destroys sessions, but does not address application passwords, which the Hardening Guide explicitly flags as a 2FA-bypassing credential type that must be managed carefully. | Add incident-response steps to enumerate and revoke application passwords for affected users. | Compare against `WordPress-Security-Hardening-Guide.md:88`, `WordPress-Security-Hardening-Guide.md:268`. |

wp-security-doc-review/rounds/2026-03-15/phase1-runbook.md

@@ -0,0 +1,15 @@
+# Phase 1 — Runbook Agent Findings
+
+Single-model, multi-agent focused review of [WP-Operations-Runbook.md](/Users/danknauss/Documents/GitHub/wordpress-runbook-template/WP-Operations-Runbook.md).
+
+| Severity | File:line | Finding | Recommendation | Verification |
+|---|---|---|---|---|
+| Critical | `WP-Operations-Runbook.md:2460-2461` | The full-restore procedure deletes the web root and raw MySQL data directory with `rm -rf`, which is unsafe, assumes a local self-managed MySQL layout, and can corrupt a running database. | Replace direct datadir deletion with a controlled restore sequence: stop services, verify target paths, recreate the database through MySQL or WP-CLI, and restore onto a clean host or verified empty target. | Rehearse Section 11.2 on staging or a disposable host and confirm the restore succeeds without touching the raw MySQL datadir. |
+| High | `WP-Operations-Runbook.md:808-830`, `WP-Operations-Runbook.md:892-937`, `WP-Operations-Runbook.md:1712-1723` | Rollback guidance is incomplete for procedures that use runtime WordPress.org updates. Database restore plus `git checkout previous-release-tag` does not reliably revert core, plugin, or theme files fetched outside the release artifact. | Require rollback from a known-good deployment artifact or filesystem backup that includes core, plugins, and themes. | In staging, perform the documented update and rollback, then verify the exact prior versions return with `wp core version`, `wp plugin list`, and `wp theme list`. |
+| High | `WP-Operations-Runbook.md:348-349`, `WP-Operations-Runbook.md:385-389`, `WP-Operations-Runbook.md:1545-1546` | Deployment workflows perform live plugin and theme updates during code deploys, which introduces uncontrolled version drift and expands blast radius during release windows. | Remove plugin and theme updates from deployment procedures and keep them in dedicated patching workflows, or require pinned artifacts that already contain the intended versions. | Compare `git rev-parse HEAD`, `wp plugin list`, and `wp theme list` before and after a staging deploy. |
+| High | `WP-Operations-Runbook.md:230`, `WP-Operations-Runbook.md:375`, `WP-Operations-Runbook.md:803`, `WP-Operations-Runbook.md:830`, `WP-Operations-Runbook.md:886`, `WP-Operations-Runbook.md:1723`, `WP-Operations-Runbook.md:3178` | Backup paths are inconsistent across the runbook: environment reference uses `/home/wordpress/backup/`, but procedures also use relative `backup/` and `backups/` paths. | Standardize on one explicit absolute backup directory outside the web root and use it consistently in backup, rollback, and quick-reference commands. | Execute the affected procedures from their stated working directories and confirm every backup is created and restored from the same location. |
+| High | `WP-Operations-Runbook.md:1386-1401`, `WP-Operations-Runbook.md:1448` | The automated backup script never sets a WordPress path or changes into the WordPress root before calling `wp db export`, which makes cron execution brittle. | Add an explicit `WP_PATH` and run `wp --path=\"$WP_PATH\" ...` or `cd \"$WP_PATH\"` before any WP-CLI command. | Run the script under a cron-like environment and confirm it can locate the installation without relying on the caller's current directory. |
+| High | `WP-Operations-Runbook.md:1418`, `WP-Operations-Runbook.md:1465`, `WP-Operations-Runbook.md:2470`, `WP-Operations-Runbook.md:2491`, `WP-Operations-Runbook.md:2583` | The backup workflow writes to an S3 site prefix, but restore commands fetch from the bucket root. Restores will miss the artifacts produced by the documented backup job. | Make the restore and verification commands use the same site prefix as the backup workflow, or redefine the object layout consistently everywhere. | Run the backup workflow, list the bucket contents, and then execute the restore commands exactly as written. |
+| High | `WP-Operations-Runbook.md:1402`, `WP-Operations-Runbook.md:2488` | The backup workflow compresses database backups to `.sql.gz`, but the full-restore procedure expects an uncompressed `.sql` file. | Make restore procedures consume `.sql.gz` directly or add an explicit decompression step before import. | Produce a backup with Section 7.2 and then run the local restore steps verbatim on staging. |
+| High | `WP-Operations-Runbook.md:2594` | `wp db reset --yes` drops all tables in the database, not just WordPress tables, which is unsafe on shared databases. | Restrict this step to dedicated WordPress databases and use a safer prefix-scoped alternative for shared databases. | Compare `wp db tables --all-tables` before recovery and confirm the reset method preserves non-WordPress tables where required. |
+| Medium | `WP-Operations-Runbook.md:129`, `WP-Operations-Runbook.md:473`, `WP-Operations-Runbook.md:2039`, `WP-Operations-Runbook.md:2456` | Several procedures assume self-managed Linux access (`ufw`, `systemctl`, direct MySQL, full filesystem control) even though the runbook still frames itself as usable for managed hosting. | Mark host-level procedures as self-managed Linux only and add managed-hosting/provider-escalation variants where equivalent access is unavailable. | Check the target environment against the required capabilities before using the procedure. |

wp-security-doc-review/rounds/2026-03-15/phase1-style-guide.md

@@ -0,0 +1,11 @@
+# Phase 1 — Style Guide Alignment Findings
+
+Focused alignment pass comparing the runbook to [WP-Security-Style-Guide.md](/Users/danknauss/Documents/GitHub/wp-security-style-guide/WP-Security-Style-Guide.md).
+
+| Severity | File:line | Finding | Recommendation | Verification |
+|---|---|---|---|---|
+| High | `WP-Operations-Runbook.md:667-693`, `WP-Operations-Runbook.md:2940` | The 2FA procedure scopes enforcement and verification to `administrator`, but the runbook also includes Multisite guidance. On Multisite, that wording can miss `Super Admin` accounts entirely. | State the scope explicitly: single-site `administrator`; Multisite `Super Admin` plus any site-level admin roles that require access. | Compare against Style Guide glossary entries for **Admin (role)** and **Super Admin**. |
+| Medium | `WP-Operations-Runbook.md:3222-3231` | The runbook glossary collapses system `cron` and `WP-Cron` into one generic definition, weakening operator clarity for Section 6.6. | Split `cron` from `WP-Cron`, and add or align glossary entries for `Dashboard`, `2FA / MFA`, `Multisite`, and `Application password` where the runbook depends on those concepts. | Compare against the Style Guide glossary entries for those terms. |
+| Medium | `WP-Operations-Runbook.md:2020` and related `admin` phrasing | The runbook still uses ambiguous `admin` wording (`admin endpoint`, `admin login`, `admin interface`) instead of the Style Guide's preferred `Dashboard` or explicit `wp-admin` path. | Replace ambiguous `admin` phrases with `Dashboard`, `Dashboard login`, or `/wp-admin/` depending on what the operator actually needs to test. | Compare against the Style Guide terminology guidance for **Dashboard** and `wp-admin`. |
+| Medium | `WP-Operations-Runbook.md:698-705`, `WP-Operations-Runbook.md:2182-2199` | The break-glass and compromise procedures do not name the Style Guide's canonical `action-gated reauthentication` pattern or require identity re-verification before sensitive identity changes. | Add an explicit requirement for action-gated reauthentication or equivalent identity re-verification before 2FA bypass, privileged password resets, user deletion, and plugin deletion. | Compare against the Style Guide glossary entry for **Action-gated reauthentication**. |
+| Medium | `WP-Operations-Runbook.md:142`, `WP-Operations-Runbook.md:3227` | `WP-CLI OAuth token` is not a canonical shared term in the Style Guide, while `Application password` is. The credential inventory may confuse operators about what secret actually exists. | Rename the row to the exact credential type in use, or make it conditional, and add an `Application password` row when relevant. | Compare against the Style Guide glossary entries for **WP-CLI** and **Application password**. |