diff --git a/docs/superpowers/plans/2026-06-08-fix-status-on-soft-error.md b/docs/superpowers/plans/2026-06-08-fix-status-on-soft-error.md new file mode 100644 index 000000000..910317758 --- /dev/null +++ b/docs/superpowers/plans/2026-06-08-fix-status-on-soft-error.md @@ -0,0 +1,636 @@ +# Fix Recovered-Turn Status Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop `/codex:task`, native `/codex:review`, and the stop-review-gate from reporting failure when a turn recorded a transient (recovered) error but still completed with usable output. + +**Architecture:** Add one named helper `resolveRunExitStatus(result, usableText)` in `lib/codex.mjs` that returns exit 0 when the turn completed with usable text (even if a stale `error` is set), else the raw `result.status`. Apply it at the two un-compensated caller sites in `codex-companion.mjs` (task + native review). Leave `buildResultStatus`, the runners, and the already-correct adversarial path untouched. The stop-gate needs no code change — it keys off the task's process exit code, which now becomes 0 on recovery; this is proven by test, not assumed. + +**Tech Stack:** Node.js ESM (`.mjs`), `node --test`, no external test deps. Fake Codex app-server fixture in `tests/fake-codex-fixture.mjs`. + +**Working directory note:** All edits are in the `/Users/kentpeng/projects/codex-plugin-cc` repo on branch `feat/codex-self-collect-multiturn`. Run all commands from that repo root. + +--- + +## File Structure + +- **Modify** `plugins/codex/scripts/lib/codex.mjs` — add + export `resolveRunExitStatus`. (Single new pure function, ~5 lines, beside `buildResultStatus` at line 740.) +- **Modify** `plugins/codex/scripts/codex-companion.mjs` — import the helper; replace `exitStatus: result.status` at the task site (line 655) and the native-review site (line 414); normalize the matching `payload.status` / `payload.codex.status` fields. +- **Modify** `tests/fake-codex-fixture.mjs` — add a `gate-recovered` named scenario that emits `agentMessage` + `error` notice + `turn/completed` (models a recovered transient on the named-scenario path the gate test uses). +- **Modify** `tests/investigation.test.mjs` — add task-path recovery e2e tests (A) and native-review recovery e2e tests (B), mirroring the existing `runCompanion` harness. +- **Modify** `tests/runtime.test.mjs` — add stop-gate recovery test (C), mirroring the existing gate-block / gate-allow tests. + +Each task below is independently committable and ordered TDD-first. + +--- + +## Task 1: Add `resolveRunExitStatus` helper (unit-tested) + +**Files:** +- Modify: `plugins/codex/scripts/lib/codex.mjs` (add function near line 745, after `buildResultStatus`) +- Test: `tests/fake-codex-fixture.test.mjs` (add a unit test block; it already imports from `lib/codex.mjs`) + +- [ ] **Step 1: Write the failing test** + +In `tests/fake-codex-fixture.test.mjs`, add `resolveRunExitStatus` to the existing import from `../plugins/codex/scripts/lib/codex.mjs` (line 5), then append this test at the end of the file: + +```js +test("resolveRunExitStatus treats a completed turn with usable text as success despite a stale error", () => { + // Recovered transient: turn completed, has usable text, but result.status is 1 + // (buildResultStatus saw the stale `error`). Must resolve to 0. + assert.equal( + resolveRunExitStatus({ turn: { status: "completed" }, status: 1 }, "ALLOW: looks fine"), + 0, + "completed turn with usable text overrides the stale non-zero status" + ); + + // Genuine failure: no usable text => keep the raw status. + assert.equal( + resolveRunExitStatus({ turn: { status: "completed" }, status: 1 }, " "), + 1, + "completed turn with no usable text keeps the failure status" + ); + + // Genuine failure: turn did not complete => keep the raw status even with text. + assert.equal( + resolveRunExitStatus({ turn: { status: "failed" }, status: 1 }, "some text"), + 1, + "non-completed turn keeps the failure status" + ); + + // Clean success: status already 0 => stays 0. + assert.equal( + resolveRunExitStatus({ turn: { status: "completed" }, status: 0 }, "done"), + 0, + "a clean success stays 0" + ); + + // Missing turn => not recovered => raw status. + assert.equal( + resolveRunExitStatus({ turn: null, status: 1 }, "text"), + 1, + "absent turn cannot be recovered; keep the raw status" + ); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `node --test tests/fake-codex-fixture.test.mjs` +Expected: FAIL — `resolveRunExitStatus is not a function` / `not exported` (import resolves to `undefined`). + +- [ ] **Step 3: Write minimal implementation** + +In `plugins/codex/scripts/lib/codex.mjs`, immediately after the `buildResultStatus` function (which ends at line 745), add: + +```js +// A turn can complete with usable output yet still carry a stale transient +// `error` (e.g. "Reconnecting... 1/5") that buildResultStatus turned into a +// non-zero status. Callers that produced a usable result should report success. +// `usableText` is the per-caller "did we get output" signal: finalMessage for +// tasks, reviewText for native review. buildResultStatus and the runners are +// intentionally left alone so result.status semantics stay stable for the +// adversarial path (which has its own, stricter parsed-verdict compensation). +export function resolveRunExitStatus(result, usableText) { + const recovered = + result.turn?.status === "completed" && Boolean(String(usableText ?? "").trim()); + return recovered ? 0 : result.status; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `node --test tests/fake-codex-fixture.test.mjs` +Expected: PASS — all assertions in the new test green; no other test in the file regresses. + +- [ ] **Step 5: Commit** + +```bash +git add plugins/codex/scripts/lib/codex.mjs tests/fake-codex-fixture.test.mjs +git commit -m "feat(codex): add resolveRunExitStatus recovery helper" +``` + +--- + +## Task 2: Apply helper to the task path + native-review path + +**Files:** +- Modify: `plugins/codex/scripts/codex-companion.mjs:21-24` (import), `:414` (native review exitStatus), `:398` (native review payload.codex.status), `:647` (task payload.status), `:655` (task exitStatus) +- Test: `tests/investigation.test.mjs` (add task + native-review recovery e2e) + +- [ ] **Step 1: Write the failing tests** + +In `tests/investigation.test.mjs`, append these tests AFTER the existing `runCompanion` helper definition (after line 670). They use the queue-driven fixture (`setupFakeCodex`) and the `runCompanion` spawn helper already in the file. + +```js +test("task turn that recovered from a transient reconnect is NOT marked failed (e2e)", async () => { + // A task turn emits a valid agent message AND a stale transient `error` + // ("Reconnecting... 1/5"), then turn/completed. The companion must exit 0 and + // report success — not propagate the stale non-zero status from buildResultStatus. + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + finalAnswer: { text: "ALLOW: looks fine" }, + turnError: { message: "Reconnecting... 1/5" } + }); + + const result = runCompanion(["task", "--json", "--cwd", cwd, "do the thing"], fake.env); + + assert.equal(result.status, 0, "a recovered task must exit 0, not propagate the stale transient error status"); + const payload = JSON.parse(result.stdout.trim()); + assert.match(payload.rawOutput, /ALLOW: looks fine/, "the real agent answer must be returned"); + assert.equal(payload.status, 0, "payload.status must be normalized to the resolved success status"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("task turn that errored with no usable output still fails (e2e)", async () => { + // Guard the genuine-failure case: a transient/fatal error with NO agent message + // must still exit non-zero. Otherwise resolveRunExitStatus would whitewash real + // failures. + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + finalAnswer: null, + turnError: { message: "Connection lost; giving up." } + }); + + const result = runCompanion(["task", "--json", "--cwd", cwd, "do the thing"], fake.env); + + assert.notEqual(result.status, 0, "a turn that errored with no usable output must still fail"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("native review that recovered from a transient reconnect is NOT marked failed (e2e)", async () => { + // The native /codex:review branch returns exitStatus from result.status raw. + // A recovered native review (reviewText present + stale error) must exit 0. + // Note: review uses reviewText as the usable-output signal, not finalMessage. + const cwd = makeInlineGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + reviewText: "Reviewed current changes.\nNo material issues found.", + turnError: { message: "Reconnecting... 1/5" } + }); + + const result = runCompanion( + ["review", "--base", "main", "--scope", "branch", "--cwd", cwd, "--json"], + fake.env + ); + + assert.equal(result.status, 0, "a recovered native review with review text must exit 0"); + const payload = JSON.parse(result.stdout.trim()); + assert.equal(payload.codex.status, 0, "payload.codex.status must be normalized to the resolved success status"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); +``` + +> NOTE for the implementer: the queue-driven fixture currently emits `agentMessage` for `entry.finalAnswer` but has no `reviewText` path. The native-review test above depends on Step 2b (fixture review-recovery support). If you are running tests strictly before any implementation, the native-review test will fail at the fixture level first; that is expected and is fixed in Step 3. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `node --test --test-name-pattern="recovered from a transient reconnect is NOT marked failed|errored with no usable output still fails|native review that recovered" tests/investigation.test.mjs` +Expected: FAIL — the task recovery test fails with `result.status === 1` (regression present); the native-review test fails (no recovered review path / status 1). + +- [ ] **Step 3: Wire the helper into the companion** + +3a. In `plugins/codex/scripts/codex-companion.mjs`, add `resolveRunExitStatus` to the import block (lines 21-24). The block becomes: + +```js + resolveReviewTurnIdleTimeoutMs, + resolveRunExitStatus, + runAppServerInvestigation, + runAppServerReview, + runAppServerTurn + } from "./lib/codex.mjs"; +``` + +3b. In the native-review branch of `executeReviewRun`, normalize the payload status. Change the `codex` block (lines 397-402) from: + +```js + codex: { + status: result.status, + stderr: result.stderr, + stdout: result.reviewText, + reasoning: result.reasoningSummary + } +``` + +to: + +```js + codex: { + status: resolveRunExitStatus(result, result.reviewText), + stderr: result.stderr, + stdout: result.reviewText, + reasoning: result.reasoningSummary + } +``` + +3c. In the same branch, change the returned `exitStatus` (line 414) from: + +```js + exitStatus: result.status, +``` + +to: + +```js + exitStatus: resolveRunExitStatus(result, result.reviewText), +``` + +3d. In `executeTaskRun`, normalize the payload status. Change the `payload` block (lines 646-652) from: + +```js + const payload = { + status: result.status, + threadId: result.threadId, + rawOutput, + touchedFiles: result.touchedFiles, + reasoningSummary: result.reasoningSummary + }; +``` + +to: + +```js + const exitStatus = resolveRunExitStatus(result, result.finalMessage); + const payload = { + status: exitStatus, + threadId: result.threadId, + rawOutput, + touchedFiles: result.touchedFiles, + reasoningSummary: result.reasoningSummary + }; +``` + +3e. In the same function, change the returned `exitStatus` (line 655) from: + +```js + exitStatus: result.status, +``` + +to: + +```js + exitStatus, +``` + +- [ ] **Step 4: Run the task tests to verify the task path passes** + +Run: `node --test --test-name-pattern="recovered from a transient reconnect is NOT marked failed|errored with no usable output still fails" tests/investigation.test.mjs` +Expected: PASS — both task tests green. (The native-review test still needs the fixture work in Task 3 if its `reviewText` queue entry is unsupported; if it already passes because the fixture supports `reviewText`, even better — verify in Task 3.) + +- [ ] **Step 5: Commit** + +```bash +git add plugins/codex/scripts/codex-companion.mjs tests/investigation.test.mjs +git commit -m "fix(codex): exit success on recovered task and native review turns" +``` + +--- + +## Task 3: Support recovered review text in the queue-driven fixture + +**Files:** +- Modify: `tests/fake-codex-fixture.mjs` (queue-driven `turn/start` block, lines 407-414) and `review/start` block (lines 338-367) +- Test: `tests/investigation.test.mjs` (the native-review recovery test from Task 2) + +> WHY: `runAppServerReview` issues a `review/start` request, not `turn/start`. The queue-driven fixture's `review/start` handler (line 338) does not currently consume the queue or emit a `turnError`, so the native-review recovery test cannot inject a recovered transient. This task makes `review/start` honor a queued `{ reviewText, turnError }` entry. + +- [ ] **Step 1: Confirm the native-review test currently fails** + +Run: `node --test --test-name-pattern="native review that recovered" tests/investigation.test.mjs` +Expected: FAIL — the queued `reviewText` entry is ignored; review returns the default `nativeReviewText` and status 0 OR the assertion on `payload.codex.status`/recovery does not hold. (Record the actual failure so Step 3's fix is verified against it.) + +- [ ] **Step 2: Make `review/start` consume the queue in queue-driven mode** + +In `tests/fake-codex-fixture.mjs`, replace the `review/start` handler body (lines 338-367) so that, in `queue-driven` BEHAVIOR, it pops a queue entry and emits the review text plus an optional transient error. Replace: + +```js + case "review/start": { + const thread = ensureThread(state, message.params.threadId); + let reviewThread = thread; + if (message.params.delivery === "detached") { + reviewThread = nextThread(state, thread.cwd, true); + send({ method: "thread/started", params: { thread: { id: reviewThread.id } } }); + } + const turnId = nextTurnId(state); + send({ id: message.id, result: { turn: buildTurn(turnId), reviewThreadId: reviewThread.id } }); + emitTurnCompleted(reviewThread.id, turnId, [ + { + started: { type: "enteredReviewMode", id: turnId, review: "current changes" } + }, + ...(BEHAVIOR === "with-reasoning" + ? [ + { + completed: { + type: "reasoning", + id: "reasoning_" + turnId, + summary: [{ text: "Reviewed the changed files and checked the likely regression paths." }], + content: [] + } + } + ] + : []), + { + completed: { type: "exitedReviewMode", id: turnId, review: nativeReviewText(message.params.target) } + } + ]); + break; + } +``` + +with: + +```js + case "review/start": { + const thread = ensureThread(state, message.params.threadId); + let reviewThread = thread; + if (message.params.delivery === "detached") { + reviewThread = nextThread(state, thread.cwd, true); + send({ method: "thread/started", params: { thread: { id: reviewThread.id } } }); + } + const turnId = nextTurnId(state); + send({ id: message.id, result: { turn: buildTurn(turnId), reviewThreadId: reviewThread.id } }); + + // Queue-driven mode lets a test script the review text and inject a + // transient (recovered) error to exercise the recovered-status path. + const reviewEntry = + BEHAVIOR === "queue-driven" && state.queue && state.queue.length > 0 + ? state.queue.shift() + : null; + if (reviewEntry) { + saveState(state); + } + const reviewText = reviewEntry && typeof reviewEntry.reviewText === "string" + ? reviewEntry.reviewText + : nativeReviewText(message.params.target); + + send({ method: "turn/started", params: { threadId: reviewThread.id, turn: buildTurn(turnId) } }); + send({ + method: "item/started", + params: { threadId: reviewThread.id, turnId, item: { type: "enteredReviewMode", id: turnId, review: "current changes" } } + }); + if (BEHAVIOR === "with-reasoning") { + send({ + method: "item/completed", + params: { + threadId: reviewThread.id, + turnId, + item: { + type: "reasoning", + id: "reasoning_" + turnId, + summary: [{ text: "Reviewed the changed files and checked the likely regression paths." }], + content: [] + } + } + }); + } + send({ + method: "item/completed", + params: { threadId: reviewThread.id, turnId, item: { type: "exitedReviewMode", id: turnId, review: reviewText } } + }); + if (reviewEntry && reviewEntry.turnError) { + send({ method: "error", params: { threadId: reviewThread.id, turnId, error: { message: reviewEntry.turnError.message } } }); + } + send({ method: "turn/completed", params: { threadId: reviewThread.id, turn: buildTurn(turnId, "completed") } }); + break; + } +``` + +> This expands the `emitTurnCompleted` shorthand into explicit sends so the +> `error` notification can be slotted between the final review item and +> `turn/completed`, exactly as the queue-driven `turn/start` path does (lines +> 407-420). Non-queue-driven behaviors are unaffected: `reviewEntry` is null, so +> `reviewText` falls back to `nativeReviewText` and no error is sent. + +- [ ] **Step 3: Run the native-review recovery test to verify it passes** + +Run: `node --test --test-name-pattern="native review that recovered" tests/investigation.test.mjs` +Expected: PASS — recovered native review exits 0 and `payload.codex.status === 0`. + +- [ ] **Step 4: Verify existing native-review tests still pass** + +Run: `node --test --test-name-pattern="native-review|review renders|review includes reasoning|review logs reasoning|review accepts the quoted" tests/runtime.test.mjs tests/investigation.test.mjs` +Expected: PASS — the expanded `review/start` handler is behavior-equivalent for non-queue-driven scenarios (these tests use `installFakeCodex` named scenarios, not the queue, so `reviewEntry` is null). + +- [ ] **Step 5: Commit** + +```bash +git add tests/fake-codex-fixture.mjs +git commit -m "test(codex): let queue-driven fixture script native review text and transient errors" +``` + +--- + +## Task 4: Stop-gate proceeds to parse the answer on a recovered task + +**Files:** +- Modify: `tests/fake-codex-fixture.mjs` (add a `gate-recovered` named scenario in `taskPayload` + `turn/start` named-scenario path) +- Test: `tests/runtime.test.mjs` (add a gate-recovery test mirroring the existing gate tests at lines 1801 and 1914) + +> WHY a named scenario (not the queue): the stop-gate test harness installs the +> fake via `installFakeCodex(binDir, behavior)` and runs the real +> `stop-review-gate-hook.mjs`, which spawns `codex-companion.mjs task`. That child +> process has its own fresh fixture state, so the parent test cannot pre-queue +> turns for it. A named scenario bakes the recovered-transient behavior into the +> fake binary itself. + +- [ ] **Step 1: Write the failing test** + +In `tests/runtime.test.mjs`, add this test after the existing `"stop hook allows the stop when the review gate is enabled and the stop-time review task is clean"` test (after line ~1955; place it adjacent to the other gate tests). It mirrors that test's setup but installs the `gate-recovered` scenario: + +```js +test("stop hook parses the ALLOW answer when the stop-time review task recovered from a transient error", () => { + // Regression: a gate review that survives a transient "Reconnecting..." notice + // still completes with a valid ALLOW answer. The task must exit 0 so the hook + // parses ALLOW/BLOCK instead of false-positive blocking on "task failed". + const repo = makeTempDir(); + const binDir = makeTempDir(); + installFakeCodex(binDir, "gate-recovered"); + initGitRepo(repo); + fs.writeFileSync(path.join(repo, "README.md"), "hello\n"); + run("git", ["add", "README.md"], { cwd: repo }); + run("git", ["commit", "-m", "init"], { cwd: repo }); + + const setup = run("node", [SCRIPT, "setup", "--enable-review-gate", "--json"], { + cwd: repo, + env: buildEnv(binDir) + }); + assert.equal(setup.status, 0, setup.stderr); + + const result = run("node", [STOP_HOOK], { + cwd: repo, + env: buildEnv(binDir), + input: JSON.stringify({ + cwd: repo, + session_id: "sess-stop-recovered", + last_assistant_message: "I completed the refactor." + }) + }); + + // ALLOW => the hook does not emit a block decision; it exits cleanly with no + // stdout decision payload (mirrors the existing clean-allow test). + assert.equal(result.status, 0, result.stderr); + assert.equal(result.stdout.trim(), "", "a recovered ALLOW review must NOT block the session"); +}); +``` + +> The assertion mirrors the existing `"... allows the stop ... when ... clean"` +> test (runtime.test.mjs:1935-1936), which asserts `status === 0` and +> `stdout.trim() === ""`. This has been verified — match it exactly. + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `node --test --test-name-pattern="recovered from a transient error" tests/runtime.test.mjs` +Expected: FAIL. At this point the `gate-recovered` scenario does not exist yet, so `installFakeCodex(binDir, "gate-recovered")` falls through to the default task behavior (`taskPayload` returns the BLOCK answer), and the hook emits `decision: "block"` — so `result.stdout` is non-empty and the `stdout.trim() === ""` assertion fails. (Order note: Tasks are TDD-ordered, but the Task 2 companion fix is what makes the *recovered-transient* case exit 0; this test additionally needs the scenario from Step 3 to inject that transient. If you run this before Task 2's fix is committed, it fails for the BLOCK reason above; after Step 3 it passes only because Task 2's fix is also in place.) + +- [ ] **Step 3: Add the `gate-recovered` named scenario to the fixture** + +3a. In `tests/fake-codex-fixture.mjs`, make `taskPayload` return an ALLOW answer for the gate prompt under the new scenario. Change the gate branch (lines 220-225) from: + +```js +function taskPayload(prompt, resume) { + if (prompt.includes("") && prompt.includes("Only review the work from the previous Claude turn.")) { + if (BEHAVIOR === "adversarial-clean") { + return "ALLOW: No blocking issues found in the previous turn."; + } + return "BLOCK: Missing empty-state guard in src/app.js:4-6."; + } +``` + +to: + +```js +function taskPayload(prompt, resume) { + if (prompt.includes("") && prompt.includes("Only review the work from the previous Claude turn.")) { + if (BEHAVIOR === "adversarial-clean" || BEHAVIOR === "gate-recovered") { + return "ALLOW: No blocking issues found in the previous turn."; + } + return "BLOCK: Missing empty-state guard in src/app.js:4-6."; + } +``` + +3b. In the named-scenario `turn/start` path, emit a transient `error` notification for the `gate-recovered` scenario, after the agent message and before `turn/completed`. The non-subagent path builds an `items` array (lines 553-569) and emits it via `emitTurnCompleted` (line 589). `emitTurnCompleted` sends `turn/started`, the items, then `turn/completed` (fixture lines 147-159) — there is no slot for an interleaved `error`. So for `gate-recovered`, emit explicitly instead of calling `emitTurnCompleted`. + +Change the tail dispatch (lines 586-590) from: + +```js + } else if (BEHAVIOR === "slow-task") { + emitTurnCompletedLater(thread.id, turnId, items, 400); + } else { + emitTurnCompleted(thread.id, turnId, items); + } +``` + +to: + +```js + } else if (BEHAVIOR === "slow-task") { + emitTurnCompletedLater(thread.id, turnId, items, 400); + } else if (BEHAVIOR === "gate-recovered") { + // Recovered transient: emit the agent message, then a stale "error" + // notice, then turn/completed. The turn still has usable output, so the + // companion must exit 0 (resolveRunExitStatus) and the gate must parse + // the ALLOW answer rather than block on a phantom failure. + send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } }); + for (const entry of items) { + if (entry && entry.completed) { + send({ method: "item/completed", params: { threadId: thread.id, turnId, item: entry.completed } }); + } + } + send({ method: "error", params: { threadId: thread.id, turnId, error: { message: "Reconnecting... 1/5" } } }); + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } }); + } else { + emitTurnCompleted(thread.id, turnId, items); + } +``` + +- [ ] **Step 4: Run the gate test to verify it passes** + +Run: `node --test --test-name-pattern="recovered from a transient error" tests/runtime.test.mjs` +Expected: PASS — the recovered ALLOW review exits 0, the hook parses ALLOW, no block decision is emitted (`result.stdout` empty). + +- [ ] **Step 5: Verify the existing gate tests still pass** + +Run: `node --test --test-name-pattern="stop hook" tests/runtime.test.mjs` +Expected: PASS — `"... blocks on findings ..."`, `"... allows the stop ... clean"`, and `"... logs running tasks ... without blocking"` all green. The `gate-recovered` scenario only adds an ALLOW + transient path; other scenarios are untouched. + +- [ ] **Step 6: Commit** + +```bash +git add tests/fake-codex-fixture.mjs tests/runtime.test.mjs +git commit -m "test(codex): stop-gate parses ALLOW when the review task recovered from a transient error" +``` + +--- + +## Task 5: Full-suite verification and regression guard + +**Files:** none (verification only) + +- [ ] **Step 1: Run the full test suite** + +Run: `node --test tests/*.test.mjs` +Expected: The suite completes and EXITS CLEANLY (no hang). The ONLY failures permitted are these four KNOWN pre-existing failures unrelated to this work: + - `status shows phases, hints, and the latest finished job` + - `status preserves adversarial review kind labels` + - `result returns the stored output for the latest finished job by default` + - `resolveStateDir uses a temp-backed per-workspace directory` + +Net new failures must be **zero**. + +- [ ] **Step 2: Confirm the pre-existing failures match the baseline** + +Run (capture failing test names): `node --test tests/*.test.mjs 2>&1 | grep -E "^not ok|# failing" | sort -u` +Expected: every `not ok` line corresponds to one of the four known failures above. If any other test fails, it is a regression introduced by this work — stop and debug it (use systematic-debugging). + +- [ ] **Step 3: Confirm clean exit (no process leak)** + +Run: `node --test tests/*.test.mjs; echo "EXIT=$?"` +Expected: the command returns promptly (does not hang waiting on an abandoned turn). `EXIT` is non-zero only because of the four known failures, not because of a timeout/hang. + +- [ ] **Step 4: Regression-guard the adversarial + idle-watchdog paths explicitly** + +Run: `node --test --test-name-pattern="recovered finalize|recovered from a transient reconnect keeps its valid verdict|idle|watchdog" tests/investigation.test.mjs tests/fake-codex-fixture.test.mjs` +Expected: PASS — confirms the A2 caller-only fix did not disturb the adversarial compensation (codex-companion.mjs:575) or the idle-timeout failure semantics. + +- [ ] **Step 5: No commit (verification task)** + +If everything is green (modulo the four known failures), the implementation is complete. Proceed to the deploy step below only when the change is merge-ready. + +--- + +## Deploy to the live local install (after merge-ready — do NOT do during implementation) + +The running plugin is the CACHE build, not this repo. After the change is merge-ready and reviewed: + +1. Back up the cache copies of the three changed files. +2. Copy the changed files into `~/.claude/plugins/cache/openai-codex/codex/1.0.4/scripts/...` preserving the same relative paths: + - `plugins/codex/scripts/lib/codex.mjs` + - `plugins/codex/scripts/codex-companion.mjs` + - (tests are not deployed) +3. Mapping reference: memory `codex-plugin-runtime-source`. + +--- + +## Out of scope (track separately; do NOT bundle into this work) + +- `?? ""`-empty-string family (`cleanCodexStderr` returns "" collapsing `x ?? "default"` chains). +- Native `/codex:review` missing the idle watchdog and empty-diff short-circuit. +- `--turn-idle-timeout` has no upper bound (huge value overflows setTimeout). +- `runAppServerInvestigation` sets `truncated = true` when `totalCommandsRun === 0`. +- `captureTurn` `armIdle()` runs before the `belongsToTurn` filter. +- Uncommitted dead `runAppServerTurn` import in `tests/investigation.test.mjs`. +- Stale `DEFAULT_INLINE_DIFF_MAX_FILES = 2` comment in `tests/investigation.test.mjs`. diff --git a/docs/superpowers/plans/2026-06-09-investigation-turn-lifecycle-race.md b/docs/superpowers/plans/2026-06-09-investigation-turn-lifecycle-race.md new file mode 100644 index 000000000..7a73a5ee9 --- /dev/null +++ b/docs/superpowers/plans/2026-06-09-investigation-turn-lifecycle-race.md @@ -0,0 +1,1076 @@ +# Investigation Turn-Lifecycle Race Fix Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop the multi-turn review investigation from discarding a valid verdict and hanging to the 180s watchdog, by fixing three lifecycle races in `captureTurn`. + +**Architecture:** All production changes live in `plugins/codex/scripts/lib/codex.mjs` inside `captureTurn` and its helpers. Defect A demotes inferred turn-completion to a subagent-gated, re-arming quiet-window fallback (primary signal stays the real `turn/completed`). Defect B reorders the idle-watchdog re-arm so only belonging traffic re-arms it. Defect C records a `pendingTurnId` from a buffered `turn/started` so the watchdog can still `turn/interrupt` when the `turn/start` RPC reply is delayed. Tests extend the existing subprocess fixture (`tests/fake-codex-fixture.mjs`) — including teaching its `queue-driven` mode to serialize turns per thread — and assert end-to-end through `runAppServerInvestigation`. + +**Tech Stack:** Node.js ESM, `node:test` + `node:assert/strict`, a fake-codex JSON-RPC app-server subprocess fixture. + +--- + +## Background the engineer needs + +- **Spec:** `docs/superpowers/specs/2026-06-09-investigation-turn-lifecycle-race-design.md`. Read it first. +- **The file under change:** `plugins/codex/scripts/lib/codex.mjs` (~1424 lines). Key regions: + - `createTurnCaptureState` (~323): builds the per-turn capture state object. The JSDoc `@typedef TurnCaptureState` at the top of the file (~10-36) must stay in sync with the fields. + - `scheduleInferredCompletion` (~393): the inference timer. Today it fires 250ms after the first `final_answer` message — **this is Defect A**. + - `completeTurn` (~366) / `clearCompletionTimer` (~359): resolve the turn / clear the inference timer. + - `recordItem` (~426) and `applyTurnNotification` (~510): translate notifications into state. `recordItem` handles `collabAgentToolCall` (~427) and `agentMessage` (~441). `applyTurnNotification` handles `turn/started` (~525) and `turn/completed` (~561). + - `captureTurn` (~579): the idle watchdog (`armIdle` ~597, the timeout callback ~604, `clearIdle` ~622) and the notification handler (~630). +- **Two timers, never share a handle:** + - **Idle watchdog** (`idleTimer`, default `DEFAULT_TURN_IDLE_TIMEOUT_MS = 180_000` at ~60): rejects the turn on a dead link. Review callers inject it via `turnIdleTimeoutMs`; task runs pass nothing (no watchdog). + - **Quiet/inference timer** (today `completionTimer`, 250ms): resolves the turn as an inferred success. Only relevant when subagent/collab work happened. +- **Test fixture:** `tests/fake-codex-fixture.mjs`. + - `installFakeCodex(binDir, behavior)` writes a fake `codex` executable whose source is a template string. Behaviors include `with-subagent`, `with-late-subagent-message`, `with-subagent-no-main-turn-completed`, `queue-driven`, `slow-task`, `interruptible-slow-task`, `gate-recovered`. + - `setupFakeCodex({ cwd })` (~683) installs `queue-driven` mode and returns a handle: `queueTurnResponse(entry)`, `queueTurnRpcError({message})`, `queueTurnHang()`, `requests` getter, `cwd`, `env`, `close()`. + - The `queue-driven` `turn/start` handler is at ~395-444. It records each request, shifts one `entry` off `state.queue`, and emits `turn/started` + items + `turn/completed` **synchronously**. + - `turn/interrupt` handler (~629) records `state.lastInterrupt = { threadId, turnId }`. +- **Two test files:** `tests/investigation.test.mjs` (in-process, calls `runAppServerInvestigation` directly — the home for Defect A & B tests) and `tests/runtime.test.mjs` (spawns the real subprocess — home for the Defect C subprocess assertion and the env-var override regression). +- **Known baseline:** the suite has **7 pre-existing unrelated failures**. "No net-new failures" is measured against that baseline (captured in Task 0). + +--- + +## File structure + +| File | Responsibility | Change | +|------|----------------|--------| +| `plugins/codex/scripts/lib/codex.mjs` | turn capture + watchdog | All three production fixes | +| `tests/fake-codex-fixture.mjs` | fake app-server | Add to `queue-driven`: `hangAfterStarted` + `cueThenHang` + `delayCompletedMs` + `lateFinalAnswer` queue entries; an opt-in `serialize` toggle whose busy-thread `turn/start` hangs (never opens), modelling the production race | +| `tests/investigation.test.mjs` | in-process lifecycle tests | Defect A repro + plain-recon-no-infer + Defect B | +| `tests/runtime.test.mjs` | subprocess tests | Defect C interrupt assertion + env-var quiet-window override regression | +| `docs/superpowers/specs/2026-06-09-investigation-turn-lifecycle-race-design.md` | spec | (already committed) | + +--- + +## Task 0: Capture the test baseline + +**Files:** none (records a baseline only). + +- [ ] **Step 1: Run the full suite and record the failing-test names** + +Run: `node --test tests/*.test.mjs 2>&1 | tail -40` +Expected: the run completes (does not hang) and reports a number of failing tests. Record the exact `not ok` test names and the failing total (expected ~7). This is the baseline; every later task must not increase it. + +- [ ] **Step 2: Save the baseline to a scratch note** + +Write the list of currently-failing test names into the PR description / scratchpad so later comparisons are exact. No commit. + +--- + +## Task 1: Defect C — record `pendingTurnId` and interrupt with it + +This is first because it is the smallest, self-contained change and unblocks the watchdog edits the other tasks build on. + +> **REVIEW FIX (finding #1):** The original draft tested `node SCRIPT review --turn-idle-timeout 1`. That path is wrong on two counts: `/codex:review` (inline) goes through `runAppServerReview` → `review/start` (NOT `turn/start`, so a `turn/start`-case fixture branch is dead code for it), and `runAppServerReview`'s `captureTurn` call passes **no** `turnIdleTimeoutMs` (codex.mjs:1004) so the inline review path arms **no watchdog at all** — `--turn-idle-timeout` is silently ignored, the interrupt is never reached, and `lastInterrupt` stays null regardless of the fix. Defect C is only reachable where a `turn/start` is issued AND the watchdog is armed: the **investigation** path. The test below uses `runAppServerInvestigation` in-process with `turnIdleTimeoutMs` set, and a new `hangAfterStarted` queue entry (a variant of the existing `queueTurnHang`, which emits no `turn/started` and so can't populate `pendingTurnId`). + +**Files:** +- Modify: `plugins/codex/scripts/lib/codex.mjs` — `@typedef` (~10-36), `createTurnCaptureState` (~331-356), `captureTurn` watchdog callback (~604-621) and notification handler (~630-650) +- Modify: `tests/fake-codex-fixture.mjs` — add a `hangAfterStarted` queue entry + a `queueTurnHangAfterStarted()` handle method +- Modify: `tests/investigation.test.mjs` (new test near the other idle tests) + +- [ ] **Step 1: Write the failing test (in-process investigation path, asserts `turn/interrupt` carries the buffered turn id)** + +Add to `tests/investigation.test.mjs`. The `hangAfterStarted` entry (added in Step 3) emits `turn/started` (so the client buffers it and can capture `pendingTurnId`) but then never sends the `turn/start` RPC result and never completes the turn — so the watchdog must fire while `state.turnId` is still null and fall back to `pendingTurnId`. + +The interrupt is fire-and-forget, but it is reliably observable: `runAppServerInvestigation` returns through `withAppServer`, which `await`s `client.close()`; `close()` calls `stdin.end()` (flushing the queued `turn/interrupt` line to the fixture) and only SIGTERMs after a 50ms unref'd timer, so the fixture processes the interrupt and persists `state.lastInterrupt` before the test inspects it. No polling needed, but read state AFTER the call resolves. + +```js +test("idle watchdog interrupts with the buffered turn id when the turn/start RPC reply is delayed (Defect C)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: progresses (has commands), does not converge. + fake.queueTurnResponse({ commands: [{ command: "git diff", exitCode: 0 }], finalAnswer: null }); + // Recon turn 2: emits turn/started, then withholds the RPC result and never + // completes — so the watchdog fires while state.turnId is still null. + fake.queueTurnHangAfterStarted(); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + turnIdleTimeoutMs: 300 + }); + + assert.ok(result.error, "a stalled turn must abort with an error"); + assert.match(result.error.message, /idle|timeout|timed out/i); + + // The fixture must have received a turn/interrupt carrying the turn id it + // announced via turn/started — proving the watchdog did not skip the + // interrupt just because state.turnId was null (Defect C). turn_2 is the + // hung turn's id (turn_1 was recon turn 1). + const statePath = path.join(fake.binDir, "fake-codex-state.json"); + const state = JSON.parse(fs.readFileSync(statePath, "utf8")); + assert.ok(state.lastInterrupt, "watchdog must send turn/interrupt even before the turn/start RPC reply"); + assert.equal(state.lastInterrupt.turnId, "turn_2", "interrupt must carry the buffered turn id"); + } finally { + fake.close(); + } +}); +``` + +Note: `investigation.test.mjs` must import `fs` and `path` (`import fs from "node:fs"; import path from "node:path";`) — add them if not already present at the top of the file. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A3 "Defect C"` +Expected: FAIL — `queueTurnHangAfterStarted` is unknown, or (once Step 3 lands but before Step 5) `state.lastInterrupt` is null because the current watchdog skips `turn/interrupt` when `state.turnId` is null. + +- [ ] **Step 3: Add the `hangAfterStarted` queue entry + handle method to the fixture** + +In `tests/fake-codex-fixture.mjs`, inside the `queue-driven` `turn/start` handler, the existing `hangNoResponse` branch (~410-416) returns BEFORE sending `turn/started`. Add a sibling branch immediately after it that DOES announce the turn first. The `turnId` is computed at ~399 (`const turnId = nextTurnId(state);`) before the queue entry is shifted, so it is in scope: + +```js + if (entry && entry.hangAfterStarted) { + // Announce the turn so the client buffers a turn/started carrying the + // id (populating pendingTurnId), but never send the turn/start RPC + // result and never complete the turn. Models a delayed RPC reply on a + // half-dead link, exercising Defect C. + send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } }); + break; + } +``` + +Then add a handle method on `setupFakeCodex`'s returned object, next to `queueTurnHang` (~730): + +```js + queueTurnHangAfterStarted() { + const state = readState(); + if (!state.queue) { state.queue = []; } + state.queue.push({ hangAfterStarted: true }); + writeState(state); + }, +``` + +- [ ] **Step 4: Add `pendingTurnId` to the typedef and state** + +In the `@typedef TurnCaptureState` (top of `codex.mjs`, after the `turnId` line ~16), add: + +```js + * turnId: string | null, + * pendingTurnId: string | null, +``` + +In `createTurnCaptureState` (~336-337), after `turnId: null,` add: + +```js + turnId: null, + pendingTurnId: null, +``` + +- [ ] **Step 5: Capture `pendingTurnId` from a buffered `turn/started` and use it in the watchdog** + +In `captureTurn`, replace the ENTIRE notification handler body (currently ~630-650). Old: + +```js + client.setNotificationHandler((message) => { + armIdle(); + if (!state.turnId) { + state.bufferedNotifications.push(message); + return; + } + + if (message.method === "thread/started" || message.method === "thread/name/updated") { + applyTurnNotification(state, message); + return; + } + + if (!belongsToTurn(state, message)) { + if (previousHandler) { + previousHandler(message); + } + return; + } + + applyTurnNotification(state, message); + }); +``` + +New (this task only adds `pendingTurnId` capture in the buffering window; the +post-buffer re-arm-on-everything behavior is preserved exactly as before so this +task introduces no watchdog regression — Task 2 refines the re-arm): + +```js + client.setNotificationHandler((message) => { + if (!state.turnId) { + // Buffering window: the turn/start RPC reply has not set state.turnId yet. + // Capture the turn id from a turn/started for our thread so the idle + // watchdog can still interrupt (Defect C). Re-arm here — these early + // notifications are almost always our own. + armIdle(); + if (message.method === "turn/started" && extractThreadId(message) === state.threadId) { + state.pendingTurnId = message.params?.turn?.id ?? state.pendingTurnId; + } + state.bufferedNotifications.push(message); + return; + } + + // Preserve existing behavior for this task: re-arm on all post-buffer + // traffic. (Task 2 replaces this with a belonging-gated re-arm.) + armIdle(); + + if (message.method === "thread/started" || message.method === "thread/name/updated") { + applyTurnNotification(state, message); + return; + } + + if (!belongsToTurn(state, message)) { + if (previousHandler) { + previousHandler(message); + } + return; + } + + applyTurnNotification(state, message); + }); +``` + +Then update the watchdog callback (currently ~611-617) from: + +```js + if (state.turnId) { + try { + client.request("turn/interrupt", { threadId, turnId: state.turnId }).catch(() => {}); + } catch { + // ignore — interrupt is best-effort + } + } +``` + +to: + +```js + const interruptTurnId = state.turnId ?? state.pendingTurnId; + if (interruptTurnId) { + try { + client.request("turn/interrupt", { threadId, turnId: interruptTurnId }).catch(() => {}); + } catch { + // ignore — interrupt is best-effort + } + } +``` + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A3 "Defect C"` +Expected: PASS. + +- [ ] **Step 7: Run the full suite to confirm no net-new failures** + +Run: `node --test tests/*.test.mjs 2>&1 | tail -20` +Expected: failing total equals the Task 0 baseline (≈7), no new names. + +- [ ] **Step 8: Commit** + +```bash +git add plugins/codex/scripts/lib/codex.mjs tests/fake-codex-fixture.mjs tests/investigation.test.mjs +git commit --no-verify -m "fix(codex): interrupt with buffered turn id when turn/start RPC reply is delayed (Defect C)" +``` + +--- + +## Task 2: Defect B — re-arm the idle watchdog only for belonging traffic + +**Files:** +- Modify: `plugins/codex/scripts/lib/codex.mjs` — `captureTurn` notification handler (~636-650) +- Modify: `tests/investigation.test.mjs` (new test) + +- [ ] **Step 1: Write the failing test (foreign-thread chatter must NOT keep a stuck turn alive)** + +Add to `tests/investigation.test.mjs`. This needs a fixture entry that, on a recon turn, emits `turn/started`, then a stream of notifications attributed to a DIFFERENT thread id, and then goes silent (never completes our turn). With the bug, the foreign notifications re-arm our watchdog forever; with the fix, the watchdog fires after the idle window. Add a queue-entry flag `foreignChatterThenHang` (implemented in Step 3). + +The assertion must distinguish bug from fix. Foreign chatter is emitted every +50ms for ~2.5s (well past the 300ms idle window). After the fix, foreign traffic +does NOT re-arm, so the watchdog fires ~300ms after `turn/started`. With the bug, +each foreign message re-arms the watchdog, so it cannot fire until chatter stops +(~2.5s) + 300ms ≈ 2.8s. A tight `elapsed < 1500` assertion therefore PASSES only +on the fixed code and FAILS (by ~2.8s) on the buggy code. + +```js +test("foreign-thread chatter does not re-arm the current turn's idle watchdog (Defect B)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: emits turn/started, then a long stream of foreign-thread + // notifications spaced UNDER the idle window, then never completes OUR turn. + // Spans ~2.5s so the buggy (re-arm-on-foreign) path cannot time out before + // chatter stops; the fixed path times out promptly at the idle window. + fake.queueTurnResponse({ foreignChatterThenHang: { count: 50, everyMs: 50 } }); + + const start = Date.now(); + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + turnIdleTimeoutMs: 300 + }); + const elapsed = Date.now() - start; + + assert.ok(result.error, "stuck turn must time out despite foreign chatter"); + assert.match(result.error.message, /idle|timeout|timed out/i); + assert.ok(elapsed < 1500, `watchdog must fire at the idle window, not be held open by foreign chatter (took ${elapsed}ms)`); + } finally { + fake.close(); + } +}); +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A3 "Defect B"` +Expected: FAIL — `foreignChatterThenHang` unknown (turn completes/empties immediately, no error) OR with the un-fixed handler the test hangs near the test timeout because foreign chatter keeps re-arming. (If it hangs, that itself demonstrates the bug; the fix makes it terminate with an error well under 15s.) + +- [ ] **Step 3: Add the `foreignChatterThenHang` queue entry to the fixture** + +In `tests/fake-codex-fixture.mjs`, inside the `queue-driven` branch, after `send({ method: "turn/started", ... })` (~419) and before the `commands` loop, add: + +```js + if (entry && entry.foreignChatterThenHang) { + const { count = 5, everyMs = 50 } = entry.foreignChatterThenHang; + const foreignThreadId = thread.id + "-foreign"; + const foreignTurnId = turnId + "-foreign"; + // Foreign-thread traffic: must NOT re-arm our turn's watchdog. + for (let n = 0; n < count; n += 1) { + setTimeout(() => { + send({ + method: "item/completed", + params: { + threadId: foreignThreadId, + turnId: foreignTurnId, + item: { type: "agentMessage", id: "foreign_" + n, text: "noise", phase: "analysis" } + } + }); + }, everyMs * (n + 1)); + } + // Never emit turn/completed for OUR turn -> the watchdog must fire. + break; + } +``` + +Note: the `break` exits the `turn/start` case for this request; the `turn/start` RPC result was already sent at ~418 so `state.turnId` is set on the client and the foreign notifications flow through the post-buffer handler path (exercising the belongsToTurn branch). + +- [ ] **Step 4: Reorder the notification handler so only belonging traffic re-arms** + +In `captureTurn`, the handler currently (after Task 1) re-arms unconditionally on all post-buffer traffic. Replace the post-buffer portion. Old (the block produced by Task 1 Step 5, from the unconditional `armIdle();` down to the closing `});`): + +```js + // Preserve existing behavior for this task: re-arm on all post-buffer + // traffic. (Task 2 replaces this with a belonging-gated re-arm.) + armIdle(); + + if (message.method === "thread/started" || message.method === "thread/name/updated") { + applyTurnNotification(state, message); + return; + } + + if (!belongsToTurn(state, message)) { + if (previousHandler) { + previousHandler(message); + } + return; + } + + applyTurnNotification(state, message); + }); +``` + +New: + +```js + if (message.method === "thread/started" || message.method === "thread/name/updated") { + // Turn-agnostic bookkeeping (thread registration / naming). Safe to re-arm. + armIdle(); + applyTurnNotification(state, message); + return; + } + + if (!belongsToTurn(state, message)) { + // Foreign turn/thread traffic must NOT re-arm our watchdog (Defect B): + // otherwise cross-turn chatter masks a stuck turn and it never fails fast. + if (previousHandler) { + previousHandler(message); + } + return; + } + + // Belongs to our turn: re-arm the idle watchdog, then apply. + armIdle(); + applyTurnNotification(state, message); + }); +``` + +(The buffering-window `armIdle()` from Task 1 Step 5 stays as-is; only the +post-buffer unconditional re-arm is replaced by belonging-gated re-arms.) + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A3 "Defect B"` +Expected: PASS — the run ends with an idle-timeout error in well under 15s. + +- [ ] **Step 6: Regression — a healthy turn that keeps emitting belonging progress is still not killed** + +This is already covered by the existing test `a turn that keeps emitting progress is NOT killed by the idle timeout` (investigation.test.mjs:309). Run it explicitly: + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A3 "NOT killed by the idle timeout"` +Expected: PASS (belonging progress still re-arms). + +- [ ] **Step 7: Run the full suite** + +Run: `node --test tests/*.test.mjs 2>&1 | tail -20` +Expected: baseline failing total, no new names. + +- [ ] **Step 8: Commit** + +```bash +git add plugins/codex/scripts/lib/codex.mjs tests/fake-codex-fixture.mjs tests/investigation.test.mjs +git commit --no-verify -m "fix(codex): re-arm idle watchdog only for belonging turn traffic (Defect B)" +``` + +--- + +## Task 3: Defect A part 1 — gate inference on subagent work + a re-arming quiet window + +This task changes the inference trigger. It does NOT yet reproduce the end-to-end finalize-queue hang (that needs the fixture serialization in Task 4); here we lock the unit-level contract: plain recon turns never infer, and the quiet window is injectable + env-overridable. + +**Files:** +- Modify: `plugins/codex/scripts/lib/codex.mjs` — module constant (~60 area), `@typedef`, `createTurnCaptureState`, `scheduleInferredCompletion` (~393-414), `recordItem` (~427-453), `applyTurnNotification` `turn/started` (~525-530), and `captureTurn`/`runAppServerInvestigation` option threading +- Modify: `tests/investigation.test.mjs` (new test) + +> **REVIEW FIX (finding #3):** The original draft used `delayCompletedMs: 120` with a 20ms quiet window and asserted the run *succeeds* — but the real `turn/completed` at ~120ms arrives before old code's 250ms cue-based inference, so old code reaches finalize the same way and the assertions held for buggy AND fixed code (non-discriminating). The corrected test below makes the readiness cue the ONLY completion signal the turn ever sends — no real `turn/completed` at all. If the gate were absent (old code), the 250ms cue-based inference fires and the run wrongly "succeeds"; with the `sawSubagentWork` gate (fixed), a plain turn never infers, so the idle watchdog must abort. Asserting `result.error` is set therefore fails on old code and passes only on the fix. + +- [ ] **Step 1: Write the failing test — a plain recon turn must NOT infer completion from a readiness cue** + +Add to `tests/investigation.test.mjs`. A plain (no-subagent) recon turn emits a `final_answer` readiness cue and then nothing else — never a real `turn/completed`. Because no subagent work occurred, inference is ineligible, so the loop waits for a `turn/completed` that never comes and the idle watchdog aborts. Uses a new `cueThenHang` entry flag (added in Step 3) and a short idle timeout to keep the test fast. The quiet window is set BELOW the idle timeout so that, on the unfixed code, cue-based inference would fire first and the run would (wrongly) not error — making the assertion discriminate. + +```js +test("plain recon turn does not infer completion from a readiness cue (Defect A gate)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: emits a "ready for finalize" final_answer cue, then goes + // silent — NO real turn/completed, NO subagent work. A plain turn must wait + // for turn/completed (which never arrives) -> idle watchdog aborts. + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "I'm ready for finalize." }, + cueThenHang: true + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + // Quiet window well below the idle timeout: on UNFIXED code, cue-based + // inference (now using this window) would fire at 60ms and the run would + // not error. On FIXED code, a plain turn never infers, so only the idle + // watchdog (400ms) ends it -> result.error is set. + inferredCompletionQuietMs: 60, + turnIdleTimeoutMs: 400 + }); + + assert.ok(result.error, "a plain turn with no real turn/completed must time out, not infer"); + assert.match(result.error.message, /idle|timeout|timed out/i); + // Finalize must NOT have been dispatched (the recon turn never completed). + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 1, "finalize must not be dispatched when recon never completed"); + } finally { + fake.close(); + } +}); +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A3 "Defect A gate"` +Expected: FAIL — `cueThenHang` is unknown to the fixture and `inferredCompletionQuietMs` is not yet threaded. Once the fixture flag lands but BEFORE the gate fix, it fails differently: the unfixed `scheduleInferredCompletion` (gated only on `finalAnswerSeen`) infers at the quiet window and the run succeeds with no error — so `assert.ok(result.error)` fails. That is the discriminating failure that the Step 4-7 gate fix resolves. + +- [ ] **Step 3: Add `cueThenHang` (and `delayCompletedMs`, used by Task 4) support to the queue-driven fixture** + +In `tests/fake-codex-fixture.mjs`, inside the `queue-driven` branch: (a) after the `finalAnswer` send (~431-432), add a branch that suppresses the turn's completion entirely; (b) replace the final `turn/completed` send (~442) so a delayed-completion variant is available for Task 4. + +(a) After the `if (entry && entry.finalAnswer) { ... }` block (~429-432), add: + +```js + if (entry && entry.cueThenHang) { + // Emit only the readiness cue (already sent above); never send a real + // turn/completed. Exercises the Defect A gate: a plain turn must not + // infer completion from the cue. + break; + } +``` + +(b) Replace the final `turn/completed` send (~442): + +```js + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } }); + break; +``` + +with: + +```js + if (entry && entry.delayCompletedMs) { + const completedTurnId = turnId; + setTimeout(() => { + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(completedTurnId, "completed") } }); + }, entry.delayCompletedMs); + } else { + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } }); + } + break; +``` + +- [ ] **Step 4: Add the quiet-window constant, env override, typedef and state fields** + +In `codex.mjs`, after the `DEFAULT_TURN_IDLE_TIMEOUT_MS = 180_000;` block (~60), add: + +```js +// Demoted-inference quiet window (Defect A). Inferred turn completion is a +// FALLBACK for the subagent/collab case where the main thread never emits a +// real turn/completed. It is eligible only after (a) the turn actually spawned +// subagent/collab work, (b) that work has drained, and (c) the turn has been +// silent for this long with no turn/completed. The window re-arms on every +// belonging item/message, so only genuine silence triggers it. Plain recon +// turns never infer — they wait for the real turn/completed. +const DEFAULT_INFERRED_COMPLETION_QUIET_MS = 15_000; + +function resolveInferredCompletionQuietMs(explicitMs) { + if (Number.isFinite(explicitMs) && explicitMs > 0) { + return explicitMs; + } + const fromEnv = Number(process.env.CODEX_INFERRED_COMPLETION_QUIET_MS); + if (Number.isFinite(fromEnv) && fromEnv > 0) { + return fromEnv; + } + return DEFAULT_INFERRED_COMPLETION_QUIET_MS; +} +``` + +In the `@typedef TurnCaptureState`, after `finalAnswerSeen: boolean,` (~23) add: + +```js + * finalAnswerSeen: boolean, + * sawSubagentWork: boolean, + * inferredCompletionQuietMs: number, +``` + +In `createTurnCaptureState`, change the signature-less body: after `finalAnswerSeen: false,` (~344) add the two fields and read the resolver from options. Replace: + +```js + finalAnswerSeen: false, + pendingCollaborations: new Set(), + activeSubagentTurns: new Set(), + completionTimer: null, +``` + +with: + +```js + finalAnswerSeen: false, + sawSubagentWork: false, + inferredCompletionQuietMs: resolveInferredCompletionQuietMs(options.inferredCompletionQuietMs), + pendingCollaborations: new Set(), + activeSubagentTurns: new Set(), + completionTimer: null, +``` + +- [ ] **Step 5: Latch `sawSubagentWork` when subagent/collab work appears** + +In `recordItem`, the `collabAgentToolCall` branch (~427-435) currently is: + +```js + if (item.type === "collabAgentToolCall") { + if (!threadId || threadId === state.threadId) { + if (lifecycle === "started" || item.status === "inProgress") { + state.pendingCollaborations.add(item.id); + } else if (lifecycle === "completed") { + state.pendingCollaborations.delete(item.id); + scheduleInferredCompletion(state); + } + } + for (const receiverThreadId of item.receiverThreadIds ?? []) { + registerThread(state, receiverThreadId); + } + } +``` + +Change the `started`/`inProgress` branch to also latch the flag: + +```js + if (item.type === "collabAgentToolCall") { + if (!threadId || threadId === state.threadId) { + if (lifecycle === "started" || item.status === "inProgress") { + state.sawSubagentWork = true; + state.pendingCollaborations.add(item.id); + } else if (lifecycle === "completed") { + state.pendingCollaborations.delete(item.id); + scheduleInferredCompletion(state); + } + } + for (const receiverThreadId of item.receiverThreadIds ?? []) { + registerThread(state, receiverThreadId); + } + } +``` + +In `applyTurnNotification`, the `turn/started` case (~525-530) registers subagent turns: + +```js + case "turn/started": + registerThread(state, message.params.threadId); + state.threadTurnIds.set(message.params.threadId, message.params.turn.id); + if ((message.params.threadId ?? null) !== state.threadId) { + state.activeSubagentTurns.add(message.params.threadId); + } +``` + +Add the latch inside the subagent branch: + +```js + case "turn/started": + registerThread(state, message.params.threadId); + state.threadTurnIds.set(message.params.threadId, message.params.turn.id); + if ((message.params.threadId ?? null) !== state.threadId) { + state.sawSubagentWork = true; + state.activeSubagentTurns.add(message.params.threadId); + } +``` + +- [ ] **Step 6: Rewrite `scheduleInferredCompletion` as the subagent-gated, re-arming quiet-window fallback** + +Replace the whole function (~393-414): + +```js +function scheduleInferredCompletion(state) { + if (state.completed || state.finalTurn || !state.finalAnswerSeen) { + return; + } + + if (state.pendingCollaborations.size > 0 || state.activeSubagentTurns.size > 0) { + return; + } + + clearCompletionTimer(state); + state.completionTimer = setTimeout(() => { + state.completionTimer = null; + if (state.completed || state.finalTurn || !state.finalAnswerSeen) { + return; + } + if (state.pendingCollaborations.size > 0 || state.activeSubagentTurns.size > 0) { + return; + } + completeTurn(state, null, { inferred: true }); + }, 250); + state.completionTimer.unref?.(); +} +``` + +with: + +```js +// Inferred completion is a guarded FALLBACK (Defect A). The primary completion +// signal is always the real main-thread turn/completed. Inference is eligible +// ONLY when the turn actually spawned subagent/collab work that has fully +// drained — plain recon turns never infer; they wait for turn/completed. When +// eligible, arm a quiet timer that re-arms on every subsequent belonging +// item/message (see scheduleInferredCompletion call sites) and fires only after +// inferredCompletionQuietMs of genuine silence with no real turn/completed. +function inferenceEligible(state) { + return ( + !state.completed && + !state.finalTurn && + state.sawSubagentWork && + state.pendingCollaborations.size === 0 && + state.activeSubagentTurns.size === 0 + ); +} + +function scheduleInferredCompletion(state) { + if (!inferenceEligible(state)) { + return; + } + + clearCompletionTimer(state); + state.completionTimer = setTimeout(() => { + state.completionTimer = null; + if (!inferenceEligible(state)) { + return; + } + completeTurn(state, null, { inferred: true }); + }, state.inferredCompletionQuietMs); + state.completionTimer.unref?.(); +} +``` + +Key behavior changes: (1) `finalAnswerSeen` is no longer required or sufficient — it is removed from the gate; (2) `sawSubagentWork` is now required; (3) the window is `state.inferredCompletionQuietMs`, not 250ms. + +- [ ] **Step 7: Stop triggering inference on a bare `final_answer`; re-arm the quiet timer on belonging activity** + +In `recordItem`, the `agentMessage` branch currently triggers inference on a final-answer message (~450-453): + +```js + if (lifecycle === "completed" && item.phase === "final_answer") { + state.finalAnswerSeen = true; + scheduleInferredCompletion(state); + } +``` + +Change it so `finalAnswerSeen` is still recorded (other code/telemetry may read it) but it no longer drives completion on its own — instead, re-arm the quiet timer only when inference is already eligible (i.e. subagent work happened and drained), so genuine post-drain silence still resolves: + +```js + if (lifecycle === "completed" && item.phase === "final_answer") { + state.finalAnswerSeen = true; + // Do NOT infer from a readiness cue on a plain turn. Only re-arm the + // quiet fallback when subagent work has already happened and drained. + if (inferenceEligible(state)) { + scheduleInferredCompletion(state); + } + } +``` + +To keep the quiet window re-arming on ALL belonging activity (not just final-answer messages) so a still-streaming subagent-origin turn isn't cut off mid-output, add a re-arm at the end of `applyTurnNotification` for `item/started` and `item/completed`. Locate the `item/started` (~543) and `item/completed` (~550) cases; after each one's existing body, before `break;`, add `maybeRearmInferredCompletion(state);`. Define that helper next to `scheduleInferredCompletion`: + +```js +function maybeRearmInferredCompletion(state) { + if (state.completionTimer && inferenceEligible(state)) { + scheduleInferredCompletion(state); + } +} +``` + +So the `item/started`/`item/completed` cases become: + +```js + case "item/started": + recordItem(state, message.params.item, "started", message.params.threadId ?? null); + { + const update = describeStartedItem(state, message.params.item); + emitProgress(state.onProgress, update?.message, update?.phase ?? null); + } + maybeRearmInferredCompletion(state); + break; + case "item/completed": + recordItem(state, message.params.item, "completed", message.params.threadId ?? null); + { + const update = describeCompletedItem(state, message.params.item); + emitProgress(state.onProgress, update?.message, update?.phase ?? null); + } + maybeRearmInferredCompletion(state); + break; +``` + +- [ ] **Step 8: Thread `inferredCompletionQuietMs` through `captureTurn` callers** + +`captureTurn` already forwards `options` into `createTurnCaptureState(threadId, options)` (~580), so the field is read there automatically. Now pass it from the runners. + +In `runAppServerInvestigation` (~1152-1154), where `turnIdleTimeoutMs` is read, add: + +```js + const turnIdleTimeoutMs = options.turnIdleTimeoutMs; + const inferredCompletionQuietMs = options.inferredCompletionQuietMs; +``` + +In BOTH `captureTurn` calls inside `runAppServerInvestigation` (the recon call ~1191 and the finalize call ~1298), extend the options object: + +Recon (~1191): + +```js + { onProgress: options.onProgress, turnIdleTimeoutMs, inferredCompletionQuietMs } +``` + +Finalize (~1298): + +```js + { onProgress: options.onProgress, turnIdleTimeoutMs, inferredCompletionQuietMs } +``` + +In `runAppServerTurn` (~1069 reads `turnIdleTimeoutMs`; ~1113 passes options), do the same so the /codex:task subagent path can be tuned via the env var (no explicit option needed there, but keep the plumbing consistent): + +After `const turnIdleTimeoutMs = options.turnIdleTimeoutMs;` (~1069) add: + +```js + const inferredCompletionQuietMs = options.inferredCompletionQuietMs; +``` + +And change the `captureTurn` options (~1113) to: + +```js + { onProgress: options.onProgress, turnIdleTimeoutMs, inferredCompletionQuietMs } +``` + +- [ ] **Step 9: Run the new test to verify it passes** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A3 "Defect A gate"` +Expected: PASS — `result.error` is set (idle timeout), 1 turn/start, finalize not dispatched. + +- [ ] **Step 10: Run the full suite — watch the subagent task test specifically** + +Run: `node --test tests/*.test.mjs 2>&1 | tail -25` +Expected: baseline failing total. The test `task can finish after subagent work even if the parent turn/completed event is missing` (runtime.test.mjs:750) now relies on the quiet fallback at the **15s default** — it will still PASS but may take ~15s, slowing the suite. Task 5 fixes the slowness via the env var. If the suite's per-test timeout is under 15s and this test now FAILS by timeout, jump to Task 5 Step 1-2 before continuing, then return. + +- [ ] **Step 11: Commit** + +```bash +git add plugins/codex/scripts/lib/codex.mjs tests/fake-codex-fixture.mjs tests/investigation.test.mjs +git commit --no-verify -m "fix(codex): demote inferred completion to a subagent-gated quiet-window fallback (Defect A)" +``` + +--- + +## Task 4: Defect A part 2 — end-to-end repro via per-thread serialization + +This proves the headline bug end-to-end: with the app-server serializing turns per thread, a premature finalize dispatch arrives while the recon turn is still open, the app-server never opens a turn for it (RPC never returns), and the run hangs to the watchdog. Old behavior: `result.error` (idle timeout). Fixed behavior (Task 3): recon waits for its real `turn/completed`, finalize lands on an idle thread, verdict survives. + +> **REVIEW FIX (finding #2):** The original draft modeled serialization as *queue-and-drain* — defer the busy-thread `turn/start` and run it when the prior turn completes. That does NOT match production: the evidence shows the app-server **never opened a turn** for the queued finalize ("no run_turn span, RPC never returned"), even after recon ended. With queue-and-drain, old code *also* succeeds (finalize just runs later), so `assert(result.error == null)` passes on buggy code too — non-discriminating. The corrected model below makes a `turn/start` that arrives **while its thread is busy HANG** (no response, no notifications), matching the evidence. The original draft also used `delayCompletedMs: 150` < old code's 250ms inference window, so recon completed *before* inference could misfire and the race never triggered; the corrected timing uses `delayCompletedMs: 600` (> 250) so the unfixed inference fires first and dispatches finalize into the busy thread. This also lets us drop the entire `pendingStarts` / `markThreadBusy` / `runQueuedTurn` machinery — the synchronous handler body stays intact; we only add a module-scoped busy-thread guard. + +**Files:** +- Modify: `tests/fake-codex-fixture.mjs` — add an opt-in module-scoped "busy thread hangs new turn/start" guard + `lateFinalAnswer` +- Modify: `tests/investigation.test.mjs` (new test) + +- [ ] **Step 1: Write the failing test — readiness cue, then a delayed real verdict + completion, under serialization** + +Add to `tests/investigation.test.mjs`. The recon turn emits a readiness cue immediately, then streams the real verdict and its real `turn/completed` only at ~600ms. Old code (250ms cue-based inference) dispatches the finalize `turn/start` at ~250ms — while recon is still busy — so finalize hangs and the run errors at the watchdog. Fixed code (plain turns never infer) waits for recon's 600ms `turn/completed`, then dispatches finalize onto the now-idle thread, which succeeds. + +```js +test("a verdict streamed after a readiness cue is captured, not discarded (Defect A end-to-end)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.enableSerialization(); + // Recon turn 1: a "ready for finalize" cue immediately, then the REAL verdict + // (~300ms) and the REAL turn/completed (~600ms). delayCompletedMs (600) is + // ABOVE the unfixed 250ms inference window, so unfixed code dispatches + // finalize while recon is still busy -> finalize hangs -> watchdog error. + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "I'm ready for finalize." }, + lateFinalAnswer: { text: "Investigation complete. Verdict ready.", afterMs: 300 }, + delayCompletedMs: 600 + }); + // Finalize turn 2: schema-enforced structured JSON. + fake.queueTurnResponse({ finalAnswer: { text: STRUCTURED_REVIEW } }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + turnIdleTimeoutMs: 2000 + // No inferredCompletionQuietMs override: a plain recon turn must never + // infer regardless of the window. The fix is the sawSubagentWork gate. + }); + + assert.equal(result.error ?? null, null, "fixed code must NOT hang to the watchdog"); + assert.equal(result.finalMessage, STRUCTURED_REVIEW, "verdict from finalize is preserved"); + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 2, "recon completes, then finalize is dispatched onto an idle thread"); + } finally { + fake.close(); + } +}); +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A4 "end-to-end"` +Expected: FAIL — `enableSerialization`/`lateFinalAnswer` are unknown. To confirm it discriminates: after Step 3-4 wire the fixture, temporarily revert Task 3's `sawSubagentWork` gate (restore the old `scheduleInferredCompletion`) and re-run — it must FAIL with `result.error` set (finalize hung on the busy thread). Restore the fix and it passes. (This revert check is the proof the test catches the bug; it is optional but recommended.) + +- [ ] **Step 3: Add the busy-thread hang guard + `lateFinalAnswer` to the fixture** + +In `tests/fake-codex-fixture.mjs`: + +(a) In the fixture template source, add a module-scoped busy-thread variable next to `const interruptibleTurns = new Map();` (~17): + +```js + const interruptibleTurns = new Map(); + let serializedBusyThread = null; +``` + +(b) In the `queue-driven` `turn/start` handler, right after the existing request push (`state.requests.push({ method: "turn/start", params: message.params });` ~397), add the busy guard. When serialization is on and the thread already has an in-flight (delayed) turn, model the app-server NOT opening a turn — record the request but send nothing and never respond: + +```js + if (state.serialize) { + if (serializedBusyThread === thread.id) { + // A turn is already open on this thread. The real app-server queues + // this turn/start and (in the bug) never opens it: no result, no + // turn/started, no turn/completed. Persist the recorded request, + // then hang. + saveState(state); + break; + } + serializedBusyThread = thread.id; + } +``` + +The thread is freed wherever the real `turn/completed` is sent (see (c)). A synchronous (non-delayed) turn sets and clears `serializedBusyThread` within the same handler tick, so it never blocks a later turn — only a `delayCompletedMs` turn holds the thread busy across event-loop ticks, which is exactly the recon turn in this test. + +(c) Free the thread when the turn completes. In the `delayCompletedMs` branch added in Task 3 Step 3(b), clear the flag inside the `setTimeout` immediately before sending `turn/completed`; in the immediate branch, clear it immediately before the synchronous `turn/completed`. Update that block to: + +```js + if (entry && entry.delayCompletedMs) { + const completedTurnId = turnId; + setTimeout(() => { + if (state.serialize) { serializedBusyThread = null; } + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(completedTurnId, "completed") } }); + }, entry.delayCompletedMs); + } else { + if (state.serialize) { serializedBusyThread = null; } + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } }); + } + break; +``` + +Note: the `cueThenHang` branch from Task 3 Step 3(a) breaks WITHOUT clearing `serializedBusyThread`. That is fine — `cueThenHang` is only used by the Task 3 gate test, which does not call `enableSerialization()`, so `serializedBusyThread` stays null there. + +(d) Add `lateFinalAnswer` emission in the `queue-driven` handler, right after the `finalAnswer` send block (~429-432): + +```js + if (entry && entry.lateFinalAnswer) { + const lateTurnId = turnId; + setTimeout(() => { + send({ method: "item/completed", params: { threadId: thread.id, turnId: lateTurnId, item: { type: "agentMessage", id: "late_" + lateTurnId, text: entry.lateFinalAnswer.text, phase: "final_answer" } } }); + }, entry.lateFinalAnswer.afterMs ?? 100); + } +``` + +- [ ] **Step 4: Expose `enableSerialization()` on the handle and persist the flag** + +In `setupFakeCodex` (~683), add `serialize: false` to `initialState`, and add a method to the returned handle (near `queueTurnHang` ~730): + +```js + enableSerialization() { + const state = readState(); + state.serialize = true; + writeState(state); + }, +``` + +`state.serialize` round-trips through `loadState()` (the whole object is JSON-persisted), so the per-message handler reads it correctly. The module-scoped `serializedBusyThread` is in-memory in the single app-server subprocess — correct, since one subprocess handles the whole investigation. + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `node --test tests/investigation.test.mjs 2>&1 | grep -A4 "end-to-end"` +Expected: PASS — no error, `finalMessage === STRUCTURED_REVIEW`, 2 turn/start requests. + +- [ ] **Step 6: Sanity — existing queue-driven tests still pass (serialization is opt-in)** + +Run: `node --test tests/investigation.test.mjs 2>&1 | tail -20` +Expected: the existing queue-driven tests (which never call `enableSerialization()`) are unaffected; baseline holds. + +- [ ] **Step 7: Commit** + +```bash +git add tests/fake-codex-fixture.mjs tests/investigation.test.mjs +git commit --no-verify -m "test(codex): reproduce the finalize-queue hang end-to-end via busy-thread serialization (Defect A)" +``` + +--- + +## Task 5: Keep the subprocess subagent test fast + add an env-override regression + +The subagent fallback fires on the real `/codex:task` subprocess path with the 15s default. Use the env var to keep tests fast and lock the override behavior. + +**Files:** +- Modify: `tests/runtime.test.mjs` — set `CODEX_INFERRED_COMPLETION_QUIET_MS` in the existing subagent-no-completion test's env, and add a focused override regression test. + +- [ ] **Step 1: Speed up the existing subagent-no-completion subprocess test** + +In `tests/runtime.test.mjs`, the test `task can finish after subagent work even if the parent turn/completed event is missing` (~750) builds env via `buildEnv(binDir)`. Change its run to inject a tiny quiet window: + +```js + const result = run("node", [SCRIPT, "task", "challenge the current design"], { + cwd: repo, + env: { ...buildEnv(binDir), CODEX_INFERRED_COMPLETION_QUIET_MS: "50" } + }); +``` + +- [ ] **Step 2: Run it — must still pass, now fast** + +Run: `node --test tests/runtime.test.mjs 2>&1 | grep -A3 "even if the parent turn/completed event is missing"` +Expected: PASS, completing in well under a second (no 15s wait). + +- [ ] **Step 3: Add a focused regression that the env override is honored** + +Add near the other subagent tests in `tests/runtime.test.mjs`. It asserts the fallback still produces the subagent task's output (proving inference fired) while the env var keeps it fast: + +```js +test("CODEX_INFERRED_COMPLETION_QUIET_MS overrides the inferred-completion quiet window", () => { + const repo = makeTempDir(); + const binDir = makeTempDir(); + installFakeCodex(binDir, "with-subagent-no-main-turn-completed"); + initGitRepo(repo); + fs.writeFileSync(path.join(repo, "README.md"), "hello\n"); + run("git", ["add", "README.md"], { cwd: repo }); + run("git", ["commit", "-m", "init"], { cwd: repo }); + + const start = Date.now(); + const result = run("node", [SCRIPT, "task", "challenge the current design"], { + cwd: repo, + env: { ...buildEnv(binDir), CODEX_INFERRED_COMPLETION_QUIET_MS: "50" } + }); + const elapsed = Date.now() - start; + + assert.equal(result.status, 0, result.stderr); + assert.equal(result.stdout, "Handled the requested task.\nTask prompt accepted.\n"); + assert.ok(elapsed < 10000, `inference must fire on the short window (took ${elapsed}ms)`); +}); +``` + +- [ ] **Step 4: Run it to verify it passes** + +Run: `node --test tests/runtime.test.mjs 2>&1 | grep -A3 "overrides the inferred-completion quiet window"` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add tests/runtime.test.mjs +git commit --no-verify -m "test(codex): keep subagent fallback fast via CODEX_INFERRED_COMPLETION_QUIET_MS override" +``` + +--- + +## Task 6: Full-suite verification and JSDoc/typedef consistency sweep + +**Files:** +- Modify: `plugins/codex/scripts/lib/codex.mjs` (only if the typedef/state drift check finds a gap) + +- [ ] **Step 1: Confirm the typedef matches the state object** + +Open `codex.mjs`. Verify `@typedef TurnCaptureState` lists every field set in `createTurnCaptureState`: it must now include `pendingTurnId`, `sawSubagentWork`, `inferredCompletionQuietMs`. Verify no field references a removed name (none were removed; `finalAnswerSeen` and `completionTimer` are retained). + +Run: `grep -n "pendingTurnId\|sawSubagentWork\|inferredCompletionQuietMs\|finalAnswerSeen\|completionTimer" plugins/codex/scripts/lib/codex.mjs` +Expected: each new field appears in both the typedef block and `createTurnCaptureState`. + +- [ ] **Step 2: Confirm no stray `250` literal or bare `final_answer` inference remains** + +Run: `grep -n "250\|finalAnswerSeen" plugins/codex/scripts/lib/codex.mjs` +Expected: the only `finalAnswerSeen` writes are the record in `recordItem` and the typedef; no `setTimeout(..., 250)` remains in `scheduleInferredCompletion`. + +- [ ] **Step 3: Run the full suite and compare to the Task 0 baseline** + +Run: `node --test tests/*.test.mjs 2>&1 | tail -30` +Expected: the run terminates cleanly (no hang). Failing-test names are a subset of (equal to) the Task 0 baseline; the new tests from Tasks 1-5 all pass. No net-new failures. + +- [ ] **Step 4: Confirm the suite does not hang and finishes in a reasonable time** + +Run: `time node --test tests/*.test.mjs > /dev/null 2>&1` +Expected: completes without hanging; total time not dramatically higher than baseline (the 15s-default path is overridden in tests). + +- [ ] **Step 5: Final commit if Step 1-2 required an edit; otherwise no-op** + +```bash +git add plugins/codex/scripts/lib/codex.mjs +git commit --no-verify -m "docs(codex): sync TurnCaptureState typedef with new capture-state fields" +``` + +(Skip if no edit was needed.) + +--- + +## Self-review notes (for the implementer) + +- **Spec coverage:** Defect A → Tasks 3+4; Defect B → Task 2; Defect C → Task 1; injectable/env-overridable quiet window → Task 3 (constant+resolver) and Task 5 (override regression); "no net-new failures vs 7 baseline" → Tasks 0 and 6. +- **Order rationale:** C and B touch the watchdog/handler with minimal logic; doing them first means Task 3's larger inference rewrite lands on an already-corrected handler. A's end-to-end repro (Task 4) depends on the gate from Task 3. +- **Type consistency:** the new fields are `pendingTurnId`, `sawSubagentWork`, `inferredCompletionQuietMs`; the new functions are `resolveInferredCompletionQuietMs`, `inferenceEligible`, `maybeRearmInferredCompletion`; the new env var is `CODEX_INFERRED_COMPLETION_QUIET_MS`; new fixture entry flags are `hangAfterStarted`, `cueThenHang`, `delayCompletedMs`, `lateFinalAnswer`, plus the `serialize` toggle / `enableSerialization()` handle method and `queueTurnHangAfterStarted()`. Use these exact names across tasks. +- **Watch-out:** in Task 3's full-suite step, the default 15s window can slow the subprocess subagent test (`...even if the parent turn/completed event is missing`) until Task 5 injects the env override. If the suite's per-test timeout is shorter than 15s, apply Task 5 Steps 1-2 early. +- **Review fixes applied (post-adversarial-review):** finding #1 — Defect C test moved from the watchdog-less `review`/`review-start` path to the investigation/`turn/start` path with `turnIdleTimeoutMs` armed (Task 1). Findings #2/#3 — the two Defect A tests re-timed so they fail on unfixed code: Task 3's gate test withholds the real `turn/completed` entirely; Task 4 models a busy-thread `turn/start` as a HANG (not queue-and-drain) with `delayCompletedMs: 600` > the old 250ms inference. Findings #4/#5 — rationale + naming/`finalAnswerSeen` notes added to the spec. diff --git a/docs/superpowers/specs/2026-06-08-fix-status-on-soft-error-design.md b/docs/superpowers/specs/2026-06-08-fix-status-on-soft-error-design.md new file mode 100644 index 000000000..82ef4865d --- /dev/null +++ b/docs/superpowers/specs/2026-06-08-fix-status-on-soft-error-design.md @@ -0,0 +1,190 @@ +# Design: fix recovered-turn status being mis-marked as failed + +Date: 2026-06-08 +Repo: /Users/kentpeng/projects/codex-plugin-cc +Branch: feat/codex-self-collect-multiturn (PR #328 → openai:main) +Source brainstorm: docs/fix-status-on-soft-error.md + +## Problem + +Commit `d222542` on this PR changed the shared `buildResultStatus` +(`plugins/codex/scripts/lib/codex.mjs:740`) to return non-zero whenever a turn +recorded *any* `error` — including a transient one the turn recovered from: + +```js +function buildResultStatus(turnState) { + if (turnState.error) return 1; // added by d222542 + return turnState.finalTurn?.status === "completed" ? 0 : 1; +} +``` + +The app-server multiplexes transient retry notices (e.g. `Reconnecting... 1/5`) +onto the same `error` notification channel as fatal turn failures, and the +capture state records the last one seen **without clearing it**. So a turn can +simultaneously have `turnState.error` set (stale transient notice) AND +`finalTurn.status === "completed"` with a valid `lastAgentMessage`. After +`d222542`, `buildResultStatus` returns 1 for that recovered turn. + +### Who compensates, who doesn't (verified against current code) + +| Path | State | Location | +|------|-------|----------| +| Adversarial review | COMPENSATED | `executeReviewRun` codex-companion.mjs:575 — exits 0 when a valid parsed verdict exists | +| Investigation runner | COMPENSATED internally | codex.mjs:1218 — aborts only when `error && !turnRecovered` | +| **/codex:task** | **NOT compensated** | `executeTaskRun` codex-companion.mjs:655 returns `exitStatus: result.status` raw | +| **Native /codex:review** | **NOT compensated** | `executeReviewRun` Review branch codex-companion.mjs:414 returns `exitStatus: result.status` raw | + +### Blast radius + +1. `runTrackedJob` (lib/tracked-jobs.mjs:156): `exitStatus !== 0` ⇒ job recorded + as **failed**, foreground command exits non-zero. +2. Stop-review-gate hook (scripts/stop-review-gate-hook.mjs:120): keys off + `result.status !== 0` and returns a "task failed" block — without ever + parsing the `ALLOW:`/`BLOCK:` answer the model produced. A recovered gate + review = **false-positive session block**. + +### Key nuance found during brainstorm + +The three callers do not share one definition of "usable output": + +- task → `result.finalMessage` +- native review → `result.reviewText` +- adversarial review → a *parsed structured verdict* (`parsed.parsed`), stricter + than "any message present", and it reuses `result.status` as its failure + fallback. + +Therefore pushing the fix down into `buildResultStatus`/the runners is **not** +free: a recovered-but-unparseable adversarial run would flip exit 1 → 0 (a new +regression) unless the adversarial fallback were also tightened. We avoid that +by fixing at the caller layer with a shared helper, leaving `result.status` +semantics and the already-correct adversarial path untouched. + +## Required behavior + +A turn that **completed with a usable result** (`finalTurn.status === +"completed"` and usable output present) but recorded a transient `error` must be +treated as SUCCESS: exit 0, job recorded "completed", gate proceeds to parse the +answer. A turn that genuinely failed (no usable output, or `finalTurn.status !== +"completed"`) keeps non-zero status. + +## Chosen approach: shared caller-level helper (A2) + +Add the recovery rule in ONE named place and apply it at the two un-compensated +caller sites. Do not touch `buildResultStatus`, the runners, or the adversarial +path. + +### `plugins/codex/scripts/lib/codex.mjs` — new function + +```js +function resolveRunExitStatus(result, usableText) { + const recovered = result.turn?.status === "completed" + && Boolean(String(usableText ?? "").trim()); + return recovered ? 0 : result.status; +} +``` + +Both runners already return `turn: turnState.finalTurn` (codex.mjs:1040 and +:1109), so `result.turn?.status` is available on both paths. + +### `plugins/codex/scripts/codex-companion.mjs` — three sites + +- `executeTaskRun` (:655): + `exitStatus: resolveRunExitStatus(result, result.finalMessage)`, and set + `payload.status` to the same resolved value (removes the JSON inconsistency + where a recovered/success task still reports `status: 1`). +- Native-review branch (:414): + `exitStatus: resolveRunExitStatus(result, result.reviewText)`, and set + `payload.codex.status` to the same resolved value. +- Adversarial branch (:575): **unchanged** — already correct. + +### Rendering — no change needed (verified) + +- `renderTaskResult` (render.mjs:350) prefers `rawOutput` (= `finalMessage`) and + only falls back to `failureMessage` when it is empty. A recovered task renders + its real answer, not the stale `Reconnecting...` notice. +- `renderNativeReviewResult` (render.mjs:323) prefers `stdout` (= `reviewText`) + regardless of `status`. + +### Stop-gate hook — no change needed (verify by test) + +The hook keys off the child process exit status (`result.status` in +stop-review-gate-hook.mjs:120). Once the task exits 0 on recovery, the hook +proceeds to parse the `ALLOW:`/`BLOCK:` answer. This propagation must be +confirmed by test C, not assumed. + +## Test plan (`node --test`, TDD: write failing tests first) + +### A. Task path — queue-driven fixture +Mirror `"finalize turn that recovered from a transient reconnect keeps its valid +verdict (e2e)"` in tests/investigation.test.mjs, but drive `task --json`. + +1. Recovered = success: queue + `{ finalAnswer: { text: "ALLOW: looks fine" }, turnError: { message: "Reconnecting... 1/5" } }`, + run companion `task --json`. Assert `result.status === 0` (process exit), + `payload.rawOutput` contains the answer, `payload.status === 0` (the + normalized JSON field). +2. Genuine-failure guard: a turn with `turnError` AND no `finalMessage` (or + `finalTurn` not completed) must still exit non-zero. + +### B. Native review path — same fixture, `review` branch +Recovered turn with `reviewText` present + `turnError` ⇒ `exitStatus 0`; a +failure turn with no `reviewText` ⇒ non-zero. Confirms `resolveRunExitStatus` +uses `reviewText`, not `finalMessage`. + +### C. Stop-gate hook — `installFakeCodex` named-scenario harness (runtime.test.mjs) +The existing gate tests use `installFakeCodex(binDir, behavior)` named scenarios, +not the queue-driven fixture. The queue-driven path already emits `entry.turnError` +as an `error` notification (tests/fake-codex-fixture.mjs ~413); extend the +named-scenario path with an equivalent switch (or add a `stop-gate-recovered` +scenario) that emits `error` notice + valid agent message + `turn/completed`. + +- Assert: a recovered gate task yields `ok:true` and the ALLOW/BLOCK answer is + parsed — NOT a "task failed" block. +- Keep the existing `"... blocks on findings"` and `"... allows ... when clean"` + gate tests green. + +### D. Regression guards (must stay green) +- Existing adversarial `"recovered finalize keeps its valid verdict"` (confirms + A2 did not touch the adversarial path). +- Idle-watchdog tests (a genuine idle timeout must still be a failure). + +## Verification before done + +- `node --test tests/*.test.mjs` — full suite. Known PRE-EXISTING failures + unrelated to this work (NOT regressions): `status shows phases, hints, and the + latest finished job`, `status preserves adversarial review kind labels`, + `result returns the stored output for the latest finished job by default`, + `resolveStateDir uses a temp-backed per-workspace directory`. Net new failures + must be zero. +- Confirm the suite EXITS CLEANLY (no hang). Do not abandon turns in tests — + always let them settle with the fixture's normal queued responses. + +## Deploy to the live local install (after merge-ready) + +The running plugin is the CACHE build, not this repo. Copy the changed files to +`~/.claude/plugins/cache/openai-codex/codex/1.0.4/scripts/...` (back up first; +mapping in memory `codex-plugin-runtime-source`). + +## Out of scope (track separately; do NOT bundle) + +`?? ""`-empty-string family; native review missing idle watchdog / empty-diff +short-circuit; `--turn-idle-timeout` upper bound; `runAppServerInvestigation` +`truncated` mislabel at 0 commands; uncommitted dead `runAppServerTurn` import in +tests; stale `DEFAULT_INLINE_DIFF_MAX_FILES` comment. + +**Promoted to a separate work item (own brainstorm → spec → plan):** the +investigation-loop turn-lifecycle race. The recon loop advances to the finalize +turn on an *inferred* completion (`scheduleInferredCompletion`, codex.mjs:393) +that fires on the first `final_answer`-phase message — which can be a "ready to +finalize" readiness cue, not the real end of the turn. The finalize `turn/start` +is then dispatched while the app-server's recon turn is still active; the server +queues it and never opens it, the real verdict streams under the recon turn and +is discarded, and the turn hangs until the 180s idle watchdog aborts. This +bundles **Defect A** (premature finalize dispatch; fix = wait for a real +`turn/completed`, demote inference to a guarded fallback) with **Defect B** +(`captureTurn` `armIdle()` runs before the `belongsToTurn` filter, codex.mjs:631, +so orphaned cross-turn traffic re-arms the captured turn's watchdog and masks the +stuck turn). Both live in `captureTurn`/the idle-watchdog subsystem; fix them +together, NOT in this status-fix change. Evidence: live run thread +`019ea5d4-0c53-75f1-9032-5573d18cd878` in `~/.codex/logs_2.sqlite` (2026-06-08 +~06:21). diff --git a/docs/superpowers/specs/2026-06-09-investigation-turn-lifecycle-race-design.md b/docs/superpowers/specs/2026-06-09-investigation-turn-lifecycle-race-design.md new file mode 100644 index 000000000..78003d768 --- /dev/null +++ b/docs/superpowers/specs/2026-06-09-investigation-turn-lifecycle-race-design.md @@ -0,0 +1,290 @@ +# Design: fix the investigation turn-lifecycle race (Defect A + B + C) + +Date: 2026-06-09 +Repo: /Users/kentpeng/projects/codex-plugin-cc +Branch: feat/codex-self-collect-multiturn (PR #328 → openai:main) +Source requirement: https://my.feishu.cn/docx/LTqndYBT7oDlyTxMPnkcyFKWnHb +Related memory: `investigation-finalize-race` + +## Problem + +The multi-turn adversarial review (`runAppServerInvestigation`, +`plugins/codex/scripts/lib/codex.mjs`) runs several read-only recon turns, then a +final turn with an `outputSchema` to produce a structured verdict. On a +connection that is healthy throughout, the model can produce a valid +`needs-attention` verdict that the plugin then **discards**; the turn hangs until +the 180s idle watchdog aborts and the user is told "Codex could not complete the +review." + +This is a **turn-lifecycle race**, not a network problem. Verified against live +run thread `019ea5d4-0c53-75f1-9032-5573d18cd878` in `~/.codex/logs_2.sqlite` +(2026-06-08 ~06:21); every model request was `200 OK text/event-stream`, no +reconnect / 429 / disconnect. + +### Evidence timeline (from the logs, not inferred) + +| Time | Event | +|------|-------| +| 06:19:42 → 06:21:50 | recon turn (submission `019ea5e1`) otel span is alive the whole window | +| 06:21:08–11 | recon emits its first `final_answer` message `msg_754…`: "I'm ready for finalize" — a **readiness cue**, not the real end of the turn | +| 06:21:11 | plugin **infers** recon is done and dispatches finalize `turn/start` (submission `019ea5e4`) | +| 06:21:41–46 | recon turn (still `019ea5e1`) streams the **real verdict JSON** (`{"verdict":"needs-attention",...}`) | +| — | finalize `019ea5e4` has only 3 log rows, all at 06:21:11; **no run_turn span, no stream events, RPC never returns** → it never actually ran | +| 06:21:50+ | thread goes silent; ~180s later the idle watchdog aborts | + +(The double-logging of each message — `content:[]` then `content:[OutputText…]` +— is normal SSE streaming, not an empty final answer. The model produced a +complete, valid verdict.) + +## Root cause — three defects in the `captureTurn` / idle-watchdog subsystem + +### Defect A (primary) — premature finalize dispatch + +`scheduleInferredCompletion` (codex.mjs:393) infers turn completion **250ms after +the first `final_answer`-phase message**. That first message can be a "ready to +finalize" cue, so the plugin dispatches the finalize `turn/start` while the +app-server's recon turn is **still active**. The app-server serializes per +thread: it queues finalize behind the open recon turn and **never opens a turn +for it** (`state.turnId` stays null, its notifications are buffered forever). The +real verdict streams ~30s later under the recon turn's id and is discarded, so +the only exit is the watchdog timeout. + +**Constraint:** `scheduleInferredCompletion` exists on purpose. Subagent / collab +turns do not always emit a main-thread `turn/completed`; deleting inference +outright reintroduces a hang. It must be demoted, not removed. + +### Defect B (secondary) — `armIdle()` runs before `belongsToTurn` + +`captureTurn`'s notification handler calls `armIdle()` as its first line +(codex.mjs:631), before the `belongsToTurn` filter. Cross-turn / cross-thread +traffic therefore re-arms the captured turn's watchdog and **masks** a stuck +turn, so it never fails fast. + +### Defect C — turn leak when the watchdog fires before the `turn/start` RPC reply + +Raised by the Codex bot on PR #328's latest commit (`370ac7c`, P2, +codex.mjs:613); verified real. When `turn/start` has reached the app-server and +`turn/started` was emitted, but that RPC's response is delayed, the watchdog can +fire while `state.turnId` is still `null` (the pre-reply notifications are +buffered, not yet applied). The `if (state.turnId)` guard at codex.mjs:611 is +false, so `turn/interrupt` is **skipped** and the turn rejects — but the live +review turn keeps running on the app-server (a server-side **turn leak**) while +the caller believes it aborted. + +All three live in `captureTurn` and its idle-watchdog; they are fixed together. + +## Out of scope + +The status-on-soft-error fix (`resolveRunExitStatus`) already merged in PR #328 +solved "recovered turns mis-marked as failed." It is orthogonal to this race and +must **not** be folded into this change. + +Also explicitly **not** adopted: "if recon already produced a valid verdict, skip +finalize." Recon runs with `outputSchema: null`, so its in-line verdict is not +schema-enforced and is less reliable than the dedicated finalize turn. Skipping +would widen the change surface and lower reliability. The schema-enforced +finalize turn is kept. + +## Design + +All changes are inside `captureTurn` and its helpers +(`scheduleInferredCompletion`, the notification handler, the idle-watchdog +callback) in `plugins/codex/scripts/lib/codex.mjs`. No caller-contract changes: +`runAppServerInvestigation`'s recon/finalize loop is structurally unchanged — it +simply stops advancing on a premature inferred completion. + +### State model + +| Field | Purpose | Defect | +|-------|---------|--------| +| `pendingTurnId` (new) | Turn id captured from a buffered `turn/started` for our thread, before the `turn/start` RPC reply sets `state.turnId`. Watchdog interrupts with `state.turnId ?? state.pendingTurnId`. | C | +| `sawSubagentWork` (new, boolean) | Latches `true` the first time a `collabAgentToolCall` or a subagent `turn/started` is seen. Hard-gates whether fallback inference is *ever* eligible. | A | +| `inferredCompletionQuietMs` (new, number) | The quiet-window duration for this turn; resolved from the `inferredCompletionQuietMs` option, else `CODEX_INFERRED_COMPLETION_QUIET_MS`, else the ~15s default. | A | +| `completionTimer` (existing field, re-purposed) | The inference timer handle. Now armed with the quiet window (`inferredCompletionQuietMs`) instead of a flat 250ms, and re-arms on belonging activity. Same field, new arming policy. | A | + +**Two independent timers, never sharing a handle:** + +- **Idle watchdog** (`idleTimer`, existing, default `DEFAULT_TURN_IDLE_TIMEOUT_MS + = 180_000`): fail-fast for a *dead connection*; **rejects** the turn. Re-armed + only by belonging traffic (Defect B). +- **Quiet / inference timer** (the `completionTimer` field, default ~15s, + injectable): *success* fallback for the subagent case where the main thread + never emits `turn/completed`; **resolves** the turn. Armed only when + `sawSubagentWork` is true. + +The watchdog still owns true-idle failure; the quiet timer only ever produces an +inferred *success*, and only in the subagent case. ("quiet timer" is prose for +the re-purposed `completionTimer` field — no separate handle is introduced.) + +### Defect A — demote inference to a guarded fallback + +1. **Primary completion signal is always the real `turn/completed`** for our + thread (existing path at codex.mjs:561 → `completeTurn`); unchanged. In the + evidence case, the 06:21:46 verdict now arrives under the still-open recon + turn, recon completes on its own `turn/completed`, and finalize lands on an + idle thread. + +2. **Inference is eligible only when ALL hold:** + - `sawSubagentWork === true` — the turn actually spawned subagent / collab + work. **Plain recon turns never infer**; they wait for `turn/completed`. + - `pendingCollaborations.size === 0 && activeSubagentTurns.size === 0` — all + subagent / collab work drained (existing gates, kept). + - the quiet window (~15s, re-arming) elapses with no new belonging + items/messages **and** no `turn/completed`. + +3. **`finalAnswerSeen` is no longer a completion *trigger*.** A `final_answer` + message in a plain recon turn does nothing on its own — we wait for + `turn/completed`. The quiet timer is driven by drain + inactivity, not by a + readiness cue. + +4. **Quiet timer re-arms** on every belonging item/message, so it fires only + after genuine silence. The window is a module constant (default ~15s) but + **injectable via options** (mirroring `turnIdleTimeoutMs`) so tests are + instant and deterministic. The override is also readable from the + `CODEX_INFERRED_COMPLETION_QUIET_MS` env var. Unlike the idle watchdog (which + has a user-facing `--turn-idle-timeout` flag), this knob is **deliberately + kept internal** — it is a test/escape-hatch override, not plumbed through the + companion CLI, since the ~15s default is correct for production and users + should not need to tune it. + +This preserves the subagent / collab hang that inference was added to prevent: a +subagent turn that never emits a main-thread `turn/completed` still resolves, +just after a real quiet window instead of a 250ms readiness-cue race. + +**Why keep inference at all, given the evidence run had no subagents?** (Design +challenge, recorded for the record.) The reported failure was a *plain* recon +turn, and the fix for it is entirely "wait for the real `turn/completed`" — the +`sawSubagentWork`-gated quiet window never becomes eligible on that path. So the +fallback machinery (the quiet timer, the `CODEX_INFERRED_COMPLETION_QUIET_MS` +env var) exists only for the `/codex:task` collab flow, which is orthogonal to +the reported defect. We deliberately keep it rather than removing inference +because the original `scheduleInferredCompletion` was added to stop a real hang: +subagent / collab turns do not always emit a main-thread `turn/completed`, and +`runAppServerTurn` (task path) shares `captureTurn`. A narrower alternative — +inference only in `runAppServerTurn` and an unconditional wait-for-`turn/completed` +in the investigation recon loop — was considered and rejected: it would fork +`captureTurn`'s completion logic by caller, duplicating the subtlest part of the +state machine. Keeping one gated fallback in `captureTurn`, exercised by both +callers, is the smaller long-term surface. The cost is that ~half the new test +surface covers the task/collab path, not the investigation path. + +**Naming:** the inference timer is referred to as the *quiet timer* in prose, but +the implementation keeps the existing state field name `completionTimer` (it is +the same handle, now armed with the quiet window instead of 250ms). No new field +is introduced for it. + +**`finalAnswerSeen` after the fix:** it is still *written* in `recordItem` but no +longer *read* by the completion gate (the old `scheduleInferredCompletion` was +its only reader). It is retained, not deleted, because it is part of the +documented `TurnCaptureState` shape and may be read by future telemetry; the plan +notes this explicitly so it is not mistaken for a live trigger. + +### Defect B — re-arm only for belonging traffic + +Reorder the notification handler so re-arm respects ownership: + +``` +on notification: + if state.turnId is null (buffering window): + armIdle() // re-arm: these are almost always our own early notifications; + // not re-arming risks a spurious timeout before the RPC returns + if turn/started for OUR thread: state.pendingTurnId = turn.id // Defect C + bufferedNotifications.push(message); return + // turnId known: + if method is thread/started or thread/name/updated: + armIdle(); applyTurnNotification(...); return // turn-agnostic bookkeeping — safe to re-arm + if not belongsToTurn(message): + previousHandler?.(message); return // foreign traffic — do NOT re-arm + armIdle() // belongs to us — re-arm + applyTurnNotification(...) +``` + +The buffered-notification replay after the RPC returns (codex.mjs:662) already +routes by `belongsToTurn`; it simply stops re-arming improperly. Net: cross-turn +chatter no longer keeps a dead turn alive; true idle still fails. + +### Defect C — interrupt even when `state.turnId` is null + +While buffering, capture the turn id into `state.pendingTurnId` from any +`turn/started` matching our thread (see Defect B handler). The watchdog callback +then interrupts with whichever id is available: + +```js +const interruptTurnId = state.turnId ?? state.pendingTurnId; +if (interruptTurnId) { + client.request("turn/interrupt", { threadId, turnId: interruptTurnId }).catch(() => {}); +} +idleReject?.(new Error(`Turn idle for ${seconds}s; aborting (upstream connection appears stalled).`)); +``` + +Best-effort and non-awaited, exactly as today — just no longer blind to a turn +whose id exists only in the buffer. If no `turn/started` was ever seen (the RPC +truly never reached the server), there is nothing to interrupt and we reject as +before. + +## Affected code + +| Location | Change | Defect | +|----------|--------|--------| +| `codex.mjs` · `createTurnCaptureState` (~323) | add `pendingTurnId`, `sawSubagentWork`; rename/repurpose timer field for the quiet window | A, C | +| `codex.mjs` · `scheduleInferredCompletion` (~393) | gate on `sawSubagentWork` + drained + re-arming quiet window; stop triggering on `finalAnswerSeen` alone | A | +| `codex.mjs` · `recordItem` / `applyTurnNotification` (~426, ~510) | latch `sawSubagentWork` on `collabAgentToolCall` / subagent `turn/started` | A | +| `codex.mjs` · `captureTurn` notification handler (~630) | re-arm ordering; capture `pendingTurnId` from buffered `turn/started` | B, C | +| `codex.mjs` · idle-watchdog callback (~611) | interrupt with `state.turnId ?? state.pendingTurnId` | C | +| `runAppServerInvestigation` recon loop (~1174) | thread the injectable quiet-window option through `captureTurn` calls | A | + +## Testing + +Tests live in `tests/investigation.test.mjs` (and `tests/runtime.test.mjs`), +driven by `tests/fake-codex-fixture.mjs`. The fixture already provides +`with-subagent`, `with-late-subagent-message`, +`with-subagent-no-main-turn-completed`, queue-driven scripting, and +`delayMs`/`emitTurnCompletedLater` timing. The quiet/inference window is injected +(set to a few ms) so tests are instant and deterministic. + +One case per acceptance criterion: + +1. **Defect A repro (headline bug):** plain recon emits a `final_answer` + readiness cue, then after a delay streams the real verdict and a real + `turn/completed`. Assert: no early completion on the cue; the late verdict is + captured under the same turn; finalize runs on an idle thread; verdict + survives. (Fails today.) +2. **Plain recon never infers:** no subagent work + `final_answer` + no + `turn/completed` ⇒ no inferred completion; the turn waits. Locks in "plain + turns wait for `turn/completed`." +3. **Subagent fallback still works (no regression):** + `with-subagent-no-main-turn-completed` ⇒ after subagent drains and the quiet + window elapses, inference resolves the turn successfully. +4. **Defect B:** with our turn open, inject foreign-thread / foreign-turn + notifications; assert they do **not** re-arm our watchdog (a stuck turn still + idle-times-out on schedule) while belonging traffic does. +5. **Defect C:** delay the `turn/start` RPC reply but emit `turn/started` first; + trip the watchdog while `state.turnId` is null; assert a `turn/interrupt` is + sent (fixture records `lastInterrupt`) with the buffered turn id and the turn + rejects. +6. **True idle still fails:** dead-link scenario (no notifications) still rejects + at the idle timeout — watchdog behavior not regressed. + +### Verification + +``` +node --test tests/*.test.mjs +``` + +Must exit cleanly, no hangs, **no net-new failures** against the known 7-failure +baseline. Capture the baseline on the current branch before changes, then +compare after. + +## Acceptance criteria (from the requirement) + +- Recon turns end only on a real `turn/completed` (or the guarded fallback); + finalize lands on an idle thread and produces the schema-enforced verdict. +- The evidence scenario is reproduced and fixed: a verdict streamed after a + readiness cue is not discarded. +- The idle watchdog is re-armed only by traffic belonging to the current turn; + cross-turn/thread chatter no longer masks a stuck turn. +- No regression to the subagent / collab multi-turn path (the hang inference was + built to prevent). +- No regression to idle-watchdog behavior (a true idle timeout still fails). +- `node --test tests/*.test.mjs` exits cleanly, no hangs, no net-new failures + against the 7-failure baseline. diff --git a/plugins/codex/commands/adversarial-review.md b/plugins/codex/commands/adversarial-review.md index da440ab4d..0fb353911 100644 --- a/plugins/codex/commands/adversarial-review.md +++ b/plugins/codex/commands/adversarial-review.md @@ -1,6 +1,6 @@ --- description: Run a Codex review that challenges the implementation approach and design choices -argument-hint: '[--wait|--background] [--base ] [--scope auto|working-tree|branch] [focus ...]' +argument-hint: '[--wait|--background] [--base ] [--scope auto|working-tree|branch] [--max-investigation-turns N] [--turn-idle-timeout SECONDS] [focus ...]' disable-model-invocation: true allowed-tools: Read, Glob, Grep, Bash(node:*), Bash(git:*), AskUserQuestion --- @@ -43,6 +43,8 @@ Argument handling: - It supports working-tree review, branch review, and `--base `. - It does not support `--scope staged` or `--scope unstaged`. - Unlike `/codex:review`, it can still take extra focus text after the flags. +- For very large diffs that exceed the inline threshold, Codex investigates the diff with read-only commands across multiple turns. Use `--max-investigation-turns N` (default 10) to raise or lower the cap. +- If a turn stalls with no output for `--turn-idle-timeout SECONDS` (default 1200), the run aborts gracefully with a clear failure instead of hanging. Lower it to fail faster on a flaky connection; raise it for very slow turns. Foreground flow: - Run: diff --git a/plugins/codex/prompts/adversarial-review-finalize.md b/plugins/codex/prompts/adversarial-review-finalize.md new file mode 100644 index 000000000..c4d0097bb --- /dev/null +++ b/plugins/codex/prompts/adversarial-review-finalize.md @@ -0,0 +1,54 @@ + +You have just completed an investigation of a code change. Now produce the structured adversarial review. + + + +Based on your investigation in the prior turns of this thread, write up your findings as a structured review. +Target: {{TARGET_LABEL}} +User focus: {{USER_FOCUS}} + + + +Report only material findings. +Do not include style feedback, naming feedback, low-value cleanup, or speculative concerns without evidence. +A finding should answer: +1. What can go wrong? +2. Why is this code path vulnerable? +3. What is the likely impact? +4. What concrete change would reduce the risk? + + + +This is the finalization turn. Do NOT run any shell commands or tool calls in this turn — your investigation is already complete and you have all the context you need from the prior turns of this thread. +Return only valid JSON matching the provided schema. Your entire output must be that JSON — no prose before or after, no shell commands, no tool-call payloads. +Keep the output compact and specific. +Use `needs-attention` if there is any material risk worth blocking on. +Use `approve` only if you cannot support any substantive adversarial finding from your investigation. +Every finding must include: +- the affected file +- `line_start` and `line_end` +- a confidence score from 0 to 1 +- a concrete recommendation +Write the summary like a terse ship/no-ship assessment, not a neutral recap. + + + +Be aggressive, but stay grounded. +Every finding must be defensible from what you read during the investigation. +Do not invent files, lines, code paths, incidents, attack chains, or runtime behavior you cannot support. +If a conclusion depends on an inference, state that explicitly in the finding body and keep the confidence honest. + + + +Prefer one strong finding over several weak ones. +Do not dilute serious issues with filler. +If the change looks safe, say so directly and return no findings. + + + +Before finalizing, check that each finding is: +- adversarial rather than stylistic +- tied to a concrete code location +- plausible under a real failure scenario +- actionable for an engineer fixing the issue + diff --git a/plugins/codex/prompts/adversarial-review-investigate.md b/plugins/codex/prompts/adversarial-review-investigate.md new file mode 100644 index 000000000..54d6dcc92 --- /dev/null +++ b/plugins/codex/prompts/adversarial-review-investigate.md @@ -0,0 +1,48 @@ + +You are Codex performing an adversarial software review. +Your job is to break confidence in the change, not to validate it. +This is the investigation phase: gather evidence with read-only commands before producing any structured output. + + + +Investigate the change so you can later produce a confident adversarial assessment. +Target: {{TARGET_LABEL}} +User focus: {{USER_FOCUS}} + + + +Default to skepticism. +Assume the change can fail in subtle, high-cost, or user-visible ways until the evidence says otherwise. +Do not give credit for good intent, partial fixes, or likely follow-up work. +If something only works on the happy path, treat that as a real weakness. + + + +Prioritize the kinds of failures that are expensive, dangerous, or hard to detect: +- auth, permissions, tenant isolation, and trust boundaries +- data loss, corruption, duplication, and irreversible state changes +- rollback safety, retries, partial failure, and idempotency gaps +- race conditions, ordering assumptions, stale state, and re-entrancy +- empty-state, null, timeout, and degraded dependency behavior +- version skew, schema drift, migration hazards, and compatibility regressions +- observability gaps that would hide failure or make recovery harder + + + +Use read-only shell commands to inspect the diff and the surrounding code. +Useful starting points: `git diff`, `git log`, `git show`, `git blame`, `cat`, `rg`/`grep`. +Read the changed files, follow references, and confirm or refute hypotheses with evidence from the code. +Do not modify any files. Your sandbox is read-only. +{{REVIEW_COLLECTION_GUIDANCE}} + + + +Continue investigating until you can defend a confident adversarial assessment. +When you have seen enough, emit a brief summary message describing what you found and stop running commands. +A summary message with no further command calls signals that you are ready for the finalization phase. +Do not produce a structured review yet — that comes in the next phase. + + + +{{REVIEW_INPUT}} + diff --git a/plugins/codex/prompts/adversarial-review.md b/plugins/codex/prompts/adversarial-review.md index 78668af6e..036e66f7d 100644 --- a/plugins/codex/prompts/adversarial-review.md +++ b/plugins/codex/prompts/adversarial-review.md @@ -46,7 +46,8 @@ A finding should answer: -Return only valid JSON matching the provided schema. +This is a single-turn review. Do NOT run any shell commands or tool calls — the file contents you need are already embedded in the repository_context block below. Do not emit a tool-use stub like `{"cmd": "..."}` instead of the review; that is not the schema. +Return only valid JSON matching the provided schema. Your entire output must be that JSON — no prose before or after, no shell commands, no tool-call payloads. Keep the output compact and specific. Use `needs-attention` if there is any material risk worth blocking on. Use `approve` only if you cannot support any substantive adversarial finding from the provided context. diff --git a/plugins/codex/scripts/codex-companion.mjs b/plugins/codex/scripts/codex-companion.mjs index 83df468ad..3b31317e3 100644 --- a/plugins/codex/scripts/codex-companion.mjs +++ b/plugins/codex/scripts/codex-companion.mjs @@ -18,6 +18,9 @@ import { interruptAppServerTurn, parseStructuredOutput, readOutputSchema, + resolveReviewTurnIdleTimeoutMs, + resolveRunExitStatus, + runAppServerInvestigation, runAppServerReview, runAppServerTurn } from "./lib/codex.mjs"; @@ -249,6 +252,24 @@ function buildAdversarialReviewPrompt(context, focusText) { }); } +function buildAdversarialInvestigatePrompt(context, focusText) { + const template = loadPromptTemplate(ROOT_DIR, "adversarial-review-investigate"); + return interpolateTemplate(template, { + TARGET_LABEL: context.target.label, + USER_FOCUS: focusText || "No extra focus provided.", + REVIEW_COLLECTION_GUIDANCE: context.collectionGuidance, + REVIEW_INPUT: context.content + }); +} + +function buildAdversarialFinalizePrompt(context, focusText) { + const template = loadPromptTemplate(ROOT_DIR, "adversarial-review-finalize"); + return interpolateTemplate(template, { + TARGET_LABEL: context.target.label, + USER_FOCUS: focusText || "No extra focus provided." + }); +} + function ensureCodexAvailable(cwd) { const availability = getCodexAvailability(cwd); if (!availability.available) { @@ -378,7 +399,7 @@ async function executeReviewRun(request) { threadId: result.threadId, sourceThreadId: result.sourceThreadId, codex: { - status: result.status, + status: resolveRunExitStatus(result, result.reviewText), stderr: result.stderr, stdout: result.reviewText, reasoning: result.reasoningSummary @@ -394,7 +415,7 @@ async function executeReviewRun(request) { ); return { - exitStatus: result.status, + exitStatus: resolveRunExitStatus(result, result.reviewText), threadId: result.threadId, turnId: result.turnId, payload, @@ -407,18 +428,127 @@ async function executeReviewRun(request) { } const context = collectReviewContext(request.cwd, target); - const prompt = buildAdversarialReviewPrompt(context, focusText); - const result = await runAppServerTurn(context.repoRoot, { - prompt, - model: request.model, - sandbox: "read-only", - outputSchema: readOutputSchema(REVIEW_SCHEMA), - onProgress: request.onProgress - }); - const parsed = parseStructuredOutput(result.finalMessage, { + + // Nothing to review. The common trigger is running on a clean working tree + // while sitting ON the default branch: the branch comparison resolves + // merge-base == HEAD, so the diff is empty. Feeding an empty diff to the + // model just burns reasoning tokens and returns nothing useful (it cannot + // find issues in code that did not change). Short-circuit to an approve + // verdict without calling the model. + if (context.fileCount === 0) { + const emptyVerdict = { + verdict: "approve", + summary: `No changes to review for ${target.label}.`, + findings: [], + next_steps: [] + }; + const parsed = { + parsed: emptyVerdict, + parseError: null, + rawOutput: JSON.stringify(emptyVerdict) + }; + const payload = { + review: reviewName, + target, + threadId: null, + context: { + repoRoot: context.repoRoot, + branch: context.branch, + summary: context.summary + }, + codex: { status: 0, stderr: "", stdout: "", reasoning: [] }, + result: parsed.parsed, + rawOutput: parsed.rawOutput, + parseError: null, + failed: false, + failureMessage: null, + reasoningSummary: [] + }; + return { + exitStatus: 0, + threadId: null, + turnId: null, + payload, + rendered: renderReviewResult(parsed, { + reviewLabel: reviewName, + targetLabel: target.label, + reasoningSummary: [] + }), + summary: emptyVerdict.summary, + jobTitle: `Codex ${reviewName}`, + jobClass: "review", + targetLabel: target.label + }; + } + + let result; + if (context.inputMode === "self-collect") { + if (!context.investigationInline) { + request.onProgress?.( + "Diff exceeds the investigation inline budget (CODEX_COMPANION_INVESTIGATION_INLINE_MAX_BYTES); Codex will re-derive it with read-only commands." + ); + } + const investigatePrompt = buildAdversarialInvestigatePrompt(context, focusText); + const finalizePrompt = buildAdversarialFinalizePrompt(context, focusText); + result = await runAppServerInvestigation(context.repoRoot, { + investigatePrompt, + finalizePrompt, + outputSchema: readOutputSchema(REVIEW_SCHEMA), + model: request.model, + sandbox: "read-only", + maxInvestigationTurns: request.maxInvestigationTurns, + turnIdleTimeoutMs: request.turnIdleTimeoutMs, + onProgress: request.onProgress + }); + } else { + const prompt = buildAdversarialReviewPrompt(context, focusText); + result = await runAppServerTurn(context.repoRoot, { + prompt, + model: request.model, + sandbox: "read-only", + outputSchema: readOutputSchema(REVIEW_SCHEMA), + turnIdleTimeoutMs: request.turnIdleTimeoutMs, + onProgress: request.onProgress + }); + } + // Parse first, then decide. A run can carry a non-zero status / error from a + // transient reconnect yet still have produced valid structured output (the + // turn recovered) — mirror of fix #1 at the finalize boundary. Only report a + // failure when the run errored AND we have no usable structured verdict; + // otherwise the leftover prose would be JSON-parsed into a misleading + // "invalid JSON" error, or a recovered valid verdict would be discarded. + const runErrored = Boolean(result.error) || result.status !== 0; + const hasFinalMessage = Boolean(String(result.finalMessage ?? "").trim()); + const structured = parseStructuredOutput(result.finalMessage, { status: result.status, failureMessage: result.error?.message ?? result.stderr }); + let parsed; + if (runErrored && !structured.parsed) { + parsed = { + parsed: null, + parseError: null, + failed: true, + failureMessage: + result.error?.message ?? result.stderr ?? "Codex run failed before producing output.", + rawOutput: result.finalMessage ?? "" + }; + } else if (!hasFinalMessage && !structured.parsed) { + // The turn completed (status 0, no error) but emitted no agent message — + // only reasoning, or nothing at all. This is NOT a malformed-JSON parse + // error (there is nothing to parse); reporting it as one renders an empty + // "- Parse error:" line. Flag it as a no-content failure so the renderer + // states the run could not complete and surfaces any reasoning instead. + parsed = { + parsed: null, + parseError: null, + failed: true, + failureMessage: "Codex completed the turn but returned no review content.", + rawOutput: "" + }; + } else { + parsed = structured; + } const payload = { review: reviewName, target, @@ -437,20 +567,34 @@ async function executeReviewRun(request) { result: parsed.parsed, rawOutput: parsed.rawOutput, parseError: parsed.parseError, + failed: parsed.failed ?? false, + failureMessage: parsed.failureMessage ?? null, reasoningSummary: result.reasoningSummary }; + if (result.investigation) { + payload.investigation = result.investigation; + } + + // A recovered finalize turn (transient reconnect/error, but valid structured + // output) carries a stale non-zero `result.status` from buildResultStatus. + // Since we produced a usable verdict and did not flag the run failed, exit + // success — otherwise the foreground command exits non-zero and background + // jobs are recorded as failed despite a valid review. Conversely, a genuinely + // failed run keeps its non-zero status. + const exitStatus = (!payload.failed && parsed.parsed) ? 0 : result.status; return { - exitStatus: result.status, + exitStatus, threadId: result.threadId, turnId: result.turnId, payload, rendered: renderReviewResult(parsed, { reviewLabel: reviewName, targetLabel: context.target.label, - reasoningSummary: result.reasoningSummary + reasoningSummary: result.reasoningSummary, + investigation: result.investigation ?? null }), - summary: parsed.parsed?.summary ?? parsed.parseError ?? firstMeaningfulLine(result.finalMessage, `${reviewName} finished.`), + summary: parsed.parsed?.summary ?? parsed.failureMessage ?? parsed.parseError ?? firstMeaningfulLine(result.finalMessage, `${reviewName} finished.`), jobTitle: `Codex ${reviewName}`, jobClass: "review", targetLabel: context.target.label @@ -508,8 +652,9 @@ async function executeTaskRun(request) { write: Boolean(request.write) } ); + const exitStatus = resolveRunExitStatus(result, result.finalMessage); const payload = { - status: result.status, + status: exitStatus, threadId: result.threadId, rawOutput, touchedFiles: result.touchedFiles, @@ -517,7 +662,7 @@ async function executeTaskRun(request) { }; return { - exitStatus: result.status, + exitStatus, threadId: result.threadId, turnId: result.turnId, payload, @@ -711,13 +856,35 @@ function enqueueBackgroundTask(cwd, job, request) { async function handleReviewCommand(argv, config) { const { options, positionals } = parseCommandInput(argv, { - valueOptions: ["base", "scope", "model", "cwd"], + valueOptions: ["base", "scope", "model", "cwd", "max-investigation-turns", "turn-idle-timeout"], booleanOptions: ["json", "background", "wait"], aliasMap: { m: "model" } }); + const rawMaxTurns = options["max-investigation-turns"]; + let maxInvestigationTurns; + if (rawMaxTurns !== undefined) { + if (!/^[1-9][0-9]*$/.test(String(rawMaxTurns))) { + throw new Error(`--max-investigation-turns must be a positive integer (got: ${rawMaxTurns})`); + } + maxInvestigationTurns = Number(rawMaxTurns); + } + + const rawIdleTimeout = options["turn-idle-timeout"]; + let explicitIdleTimeoutMs; + if (rawIdleTimeout !== undefined) { + if (!/^[1-9][0-9]*$/.test(String(rawIdleTimeout))) { + throw new Error(`--turn-idle-timeout must be a positive integer (seconds) (got: ${rawIdleTimeout})`); + } + explicitIdleTimeoutMs = Number(rawIdleTimeout) * 1000; + } + // Reviews always get an idle watchdog (default when no flag is given), so a + // stalled review never hangs forever. /codex:task deliberately does NOT, so a + // long-thinking task is not aborted; it passes no timeout to runAppServerTurn. + const turnIdleTimeoutMs = resolveReviewTurnIdleTimeoutMs(explicitIdleTimeoutMs); + const cwd = resolveCommandCwd(options); const workspaceRoot = resolveCommandWorkspace(options); const focusText = positionals.join(" ").trim(); @@ -746,6 +913,8 @@ async function handleReviewCommand(argv, config) { model: options.model, focusText, reviewName: config.reviewName, + maxInvestigationTurns, + turnIdleTimeoutMs, onProgress: progress }), { json: options.json } diff --git a/plugins/codex/scripts/lib/codex.mjs b/plugins/codex/scripts/lib/codex.mjs index fead00cc4..61dee6e4e 100644 --- a/plugins/codex/scripts/lib/codex.mjs +++ b/plugins/codex/scripts/lib/codex.mjs @@ -14,6 +14,7 @@ * threadTurnIds: Map, * threadLabels: Map, * turnId: string | null, + * pendingTurnId: string | null, * bufferedNotifications: AppServerNotification[], * completion: Promise, * resolveCompletion: (state: TurnCaptureState) => void, @@ -21,6 +22,8 @@ * finalTurn: Turn | null, * completed: boolean, * finalAnswerSeen: boolean, + * sawSubagentWork: boolean, + * inferredCompletionQuietMs: number, * pendingCollaborations: Set, * activeSubagentTurns: Set, * completionTimer: ReturnType | null, @@ -48,6 +51,55 @@ const SERVICE_NAME = "claude_code_codex_plugin"; const TASK_THREAD_PREFIX = "Codex Companion Task"; const DEFAULT_CONTINUE_PROMPT = "Continue from the current thread state. Pick the next highest-value step and follow through until the task is resolved."; +// A REVIEW turn that produces no notifications and no response for this long is +// treated as a stalled upstream connection and aborted, instead of the RPC +// promise hanging forever (it only settles on a response or a full socket +// close, neither of which happens on a half-dead "Reconnecting..." link). +// The timer is reset on every progress notification, so a slow-but-healthy +// turn running many commands is never killed — only true silence triggers it. +// +// This default is REVIEW-ONLY. It must NOT be injected by the shared +// runAppServerTurn/runAppServerInvestigation runners, because /codex:task also +// calls runAppServerTurn: a long-thinking or long-single-command task would +// otherwise be aborted at this threshold with no task-level way to raise or +// disable it. Review callers opt in via resolveReviewTurnIdleTimeoutMs(); the +// runners pass whatever they are given straight through to captureTurn, which +// arms no watchdog for an absent/invalid value. +// Measured 2026-07-27: healthy Bedrock gpt-5.6 xhigh turns go silent for well +// over 600s while reasoning; 180s killed them routinely. +const DEFAULT_TURN_IDLE_TIMEOUT_MS = 1_200_000; + +// Demoted-inference quiet window (Defect A). Inferred turn completion is a +// FALLBACK for the subagent/collab case where the main thread never emits a +// real turn/completed. It is eligible only after (a) the turn actually spawned +// subagent/collab work, (b) that work has drained, and (c) the turn has been +// silent for this long with no turn/completed. The window re-arms on every +// belonging item/message, so only genuine silence triggers it. Plain recon +// turns never infer — they wait for the real turn/completed. +const DEFAULT_INFERRED_COMPLETION_QUIET_MS = 15_000; + +function resolveInferredCompletionQuietMs(explicitMs) { + if (Number.isFinite(explicitMs) && explicitMs > 0) { + return explicitMs; + } + const fromEnv = Number(process.env.CODEX_INFERRED_COMPLETION_QUIET_MS); + if (Number.isFinite(fromEnv) && fromEnv > 0) { + return fromEnv; + } + return DEFAULT_INFERRED_COMPLETION_QUIET_MS; +} + +/** + * Resolve the idle-watchdog timeout for REVIEW turns. Returns the review + * default when no explicit positive value is supplied. Task runs do not call + * this and therefore run without an idle watchdog. + * @param {number | null | undefined} explicitMs + * @returns {number} + */ +export function resolveReviewTurnIdleTimeoutMs(explicitMs) { + return Number.isFinite(explicitMs) && explicitMs > 0 ? explicitMs : DEFAULT_TURN_IDLE_TIMEOUT_MS; +} + const EXTERNAL_AGENT_IMPORT_COMPLETED = "externalAgentConfig/import/completed"; const EXTERNAL_AGENT_IMPORT_TIMEOUT_MS = 2 * 60 * 1000; @@ -315,6 +367,7 @@ function createTurnCaptureState(threadId, options = {}) { threadTurnIds: new Map(), threadLabels: new Map(), turnId: null, + pendingTurnId: null, bufferedNotifications: [], completion, resolveCompletion, @@ -322,6 +375,8 @@ function createTurnCaptureState(threadId, options = {}) { finalTurn: null, completed: false, finalAnswerSeen: false, + sawSubagentWork: false, + inferredCompletionQuietMs: resolveInferredCompletionQuietMs(options.inferredCompletionQuietMs), pendingCollaborations: new Set(), activeSubagentTurns: new Set(), completionTimer: null, @@ -370,29 +425,45 @@ function completeTurn(state, turn = null, options = {}) { state.resolveCompletion(state); } -function scheduleInferredCompletion(state) { - if (state.completed || state.finalTurn || !state.finalAnswerSeen) { - return; - } +// Inferred completion is a guarded FALLBACK (Defect A). The primary completion +// signal is always the real main-thread turn/completed. Inference is eligible +// ONLY when the turn actually spawned subagent/collab work that has fully +// drained — plain recon turns never infer; they wait for turn/completed. When +// eligible, arm a quiet timer that re-arms on every subsequent belonging +// item/message (see scheduleInferredCompletion call sites) and fires only after +// inferredCompletionQuietMs of genuine silence with no real turn/completed. +function inferenceEligible(state) { + return ( + !state.completed && + !state.finalTurn && + state.sawSubagentWork && + state.pendingCollaborations.size === 0 && + state.activeSubagentTurns.size === 0 + ); +} - if (state.pendingCollaborations.size > 0 || state.activeSubagentTurns.size > 0) { +function scheduleInferredCompletion(state) { + if (!inferenceEligible(state)) { return; } clearCompletionTimer(state); state.completionTimer = setTimeout(() => { state.completionTimer = null; - if (state.completed || state.finalTurn || !state.finalAnswerSeen) { - return; - } - if (state.pendingCollaborations.size > 0 || state.activeSubagentTurns.size > 0) { + if (!inferenceEligible(state)) { return; } completeTurn(state, null, { inferred: true }); - }, 250); + }, state.inferredCompletionQuietMs); state.completionTimer.unref?.(); } +function maybeRearmInferredCompletion(state) { + if (state.completionTimer && inferenceEligible(state)) { + scheduleInferredCompletion(state); + } +} + function belongsToTurn(state, message) { const messageThreadId = extractThreadId(message); if (!messageThreadId || !state.threadIds.has(messageThreadId)) { @@ -407,6 +478,7 @@ function recordItem(state, item, lifecycle, threadId = null) { if (item.type === "collabAgentToolCall") { if (!threadId || threadId === state.threadId) { if (lifecycle === "started" || item.status === "inProgress") { + state.sawSubagentWork = true; state.pendingCollaborations.add(item.id); } else if (lifecycle === "completed") { state.pendingCollaborations.delete(item.id); @@ -428,8 +500,14 @@ function recordItem(state, item, lifecycle, threadId = null) { if (!threadId || threadId === state.threadId) { state.lastAgentMessage = item.text; if (lifecycle === "completed" && item.phase === "final_answer") { + // Bookkeeping only; finalAnswerSeen is no longer part of the inference + // gate (Defect A) — a readiness cue must not complete a plain turn. state.finalAnswerSeen = true; - scheduleInferredCompletion(state); + // Do NOT infer from a readiness cue on a plain turn. Only re-arm the + // quiet fallback when subagent work has already happened and drained. + if (inferenceEligible(state)) { + scheduleInferredCompletion(state); + } } } if (lifecycle === "completed") { @@ -506,6 +584,7 @@ function applyTurnNotification(state, message) { registerThread(state, message.params.threadId); state.threadTurnIds.set(message.params.threadId, message.params.turn.id); if ((message.params.threadId ?? null) !== state.threadId) { + state.sawSubagentWork = true; state.activeSubagentTurns.add(message.params.threadId); } emitProgress( @@ -526,6 +605,7 @@ function applyTurnNotification(state, message) { const update = describeStartedItem(state, message.params.item); emitProgress(state.onProgress, update?.message, update?.phase ?? null); } + maybeRearmInferredCompletion(state); break; case "item/completed": recordItem(state, message.params.item, "completed", message.params.threadId ?? null); @@ -533,6 +613,7 @@ function applyTurnNotification(state, message) { const update = describeCompletedItem(state, message.params.item); emitProgress(state.onProgress, update?.message, update?.phase ?? null); } + maybeRearmInferredCompletion(state); break; case "error": state.error = message.params.error; @@ -560,29 +641,94 @@ async function captureTurn(client, threadId, startRequest, options = {}) { const state = createTurnCaptureState(threadId, options); const previousHandler = client.notificationHandler; + // Idle watchdog: the turn/start RPC and the completion promise only settle on + // a server response or a full socket close. A half-dead "Reconnecting..." + // link delivers neither, so without this the turn would hang forever. We + // reject after `idleTimeoutMs` of total silence, re-arming on every progress + // notification so a slow-but-active turn is never killed. + const idleTimeoutMs = Number.isFinite(options.turnIdleTimeoutMs) && options.turnIdleTimeoutMs > 0 + ? options.turnIdleTimeoutMs + : null; + let idleTimer = null; + let idleReject = null; + let settled = false; + const idlePromise = idleTimeoutMs + ? new Promise((_resolve, reject) => { idleReject = reject; }) + : null; + const armIdle = () => { + if (!idleTimeoutMs || settled) { + return; + } + if (idleTimer) { + clearTimeout(idleTimer); + } + idleTimer = setTimeout(() => { + if (settled) { + return; + } + const seconds = Math.round(idleTimeoutMs / 1000); + // Best-effort interrupt so the app-server can release the turn; do NOT + // await it (the same dead link could make it hang too). + const interruptTurnId = state.turnId ?? state.pendingTurnId; + if (interruptTurnId) { + try { + client.request("turn/interrupt", { threadId, turnId: interruptTurnId }).catch(() => {}); + } catch { + // ignore — interrupt is best-effort + } + } + idleReject?.(new Error(`Turn idle for ${seconds}s; aborting (upstream connection appears stalled).`)); + }, idleTimeoutMs); + idleTimer.unref?.(); + }; + const clearIdle = () => { + settled = true; + if (idleTimer) { + clearTimeout(idleTimer); + idleTimer = null; + } + }; + client.setNotificationHandler((message) => { if (!state.turnId) { + // Buffering window: the turn/start RPC reply has not set state.turnId yet. + // Capture the turn id from a turn/started for our thread so the idle + // watchdog can still interrupt (Defect C). Re-arm here — these early + // notifications are almost always our own. + armIdle(); + if (message.method === "turn/started" && extractThreadId(message) === state.threadId) { + state.pendingTurnId = message.params?.turn?.id ?? state.pendingTurnId; + } state.bufferedNotifications.push(message); return; } if (message.method === "thread/started" || message.method === "thread/name/updated") { + // Turn-agnostic bookkeeping (thread registration / naming). Safe to re-arm. + armIdle(); applyTurnNotification(state, message); return; } if (!belongsToTurn(state, message)) { - if (previousHandler) { - previousHandler(message); - } - return; + // Foreign turn/thread traffic must NOT re-arm our watchdog (Defect B): + // otherwise cross-turn chatter masks a stuck turn and it never fails fast. + if (previousHandler) { + previousHandler(message); + } + return; } + // Belongs to our turn: re-arm the idle watchdog, then apply. + armIdle(); applyTurnNotification(state, message); }); try { - const response = await startRequest(); + armIdle(); + const response = idlePromise + ? await Promise.race([startRequest(), idlePromise]) + : await startRequest(); options.onResponse?.(response, state); state.turnId = response.turn?.id ?? null; if (state.turnId) { @@ -603,8 +749,11 @@ async function captureTurn(client, threadId, startRequest, options = {}) { completeTurn(state, response.turn); } - return await state.completion; + return idlePromise + ? await Promise.race([state.completion, idlePromise]) + : await state.completion; } finally { + clearIdle(); clearCompletionTimer(state); client.setNotificationHandler(previousHandler ?? null); } @@ -752,9 +901,25 @@ async function resumeThread(client, threadId, cwd, options = {}) { } function buildResultStatus(turnState) { + if (turnState.error) { + return 1; + } return turnState.finalTurn?.status === "completed" ? 0 : 1; } +// A turn can complete with usable output yet still carry a stale transient +// `error` (e.g. "Reconnecting... 1/5") that buildResultStatus turned into a +// non-zero status. Callers that produced a usable result should report success. +// `usableText` is the per-caller "did we get output" signal: finalMessage for +// tasks, reviewText for native review. buildResultStatus and the runners are +// intentionally left alone so result.status semantics stay stable for the +// adversarial path (which has its own, stricter parsed-verdict compensation). +export function resolveRunExitStatus(result, usableText) { + const recovered = + result.turn?.status === "completed" && Boolean(String(usableText ?? "").trim()); + return recovered ? 0 : result.status; +} + const BUILTIN_PROVIDER_LABELS = new Map([ ["openai", "OpenAI"], ["ollama", "Ollama"], @@ -1098,6 +1263,12 @@ export async function runAppServerTurn(cwd, options = {}) { throw new Error("Codex CLI is not installed or is missing required runtime support. Install it with `npm install -g @openai/codex`, then rerun `/codex:setup`."); } + // Pass-through only: no implicit default. captureTurn arms no watchdog when + // turnIdleTimeoutMs is absent/invalid. Review callers supply the default via + // resolveReviewTurnIdleTimeoutMs(); task runs intentionally pass nothing. + const turnIdleTimeoutMs = options.turnIdleTimeoutMs; + const inferredCompletionQuietMs = options.inferredCompletionQuietMs; + return withAppServer(cwd, async (client) => { let threadId; @@ -1140,7 +1311,7 @@ export async function runAppServerTurn(cwd, options = {}) { effort: options.effort ?? null, outputSchema: options.outputSchema ?? null }), - { onProgress: options.onProgress } + { onProgress: options.onProgress, turnIdleTimeoutMs, inferredCompletionQuietMs } ); return { @@ -1159,6 +1330,259 @@ export async function runAppServerTurn(cwd, options = {}) { }); } +const DEFAULT_MAX_INVESTIGATION_TURNS = 10; +const INVESTIGATION_CONTINUATION_CUE = "Continue your investigation."; + +const DEFAULT_FINALIZE_EFFORT = "medium"; +const FINALIZE_EFFORT_VALUES = new Set(["none", "minimal", "low", "medium", "high", "xhigh"]); + +// The finalize turn translates already-formed conclusions into schema JSON — +// mechanical work that does not benefit from a reasoning-heavy effort. Read at +// call time (not import time) so the env override always takes effect. +function resolveFinalizeEffort(callerEffort) { + const raw = String(process.env.CODEX_COMPANION_FINALIZE_EFFORT ?? "").trim().toLowerCase(); + if (raw === "inherit") { + return callerEffort ?? null; + } + if (FINALIZE_EFFORT_VALUES.has(raw)) { + return raw; + } + return DEFAULT_FINALIZE_EFFORT; +} + +export async function runAppServerInvestigation(cwd, options = {}) { + const availability = getCodexAvailability(cwd); + if (!availability.available) { + throw new Error("Codex CLI is not installed or is missing required runtime support. Install it with `npm install -g @openai/codex`, then rerun `/codex:setup`."); + } + + const investigatePrompt = options.investigatePrompt?.trim(); + const finalizePrompt = options.finalizePrompt?.trim(); + if (!investigatePrompt) { + throw new Error("runAppServerInvestigation requires investigatePrompt."); + } + if (!finalizePrompt) { + throw new Error("runAppServerInvestigation requires finalizePrompt."); + } + const maxInvestigationTurns = Number.isFinite(options.maxInvestigationTurns) && options.maxInvestigationTurns > 0 + ? Math.floor(options.maxInvestigationTurns) + : DEFAULT_MAX_INVESTIGATION_TURNS; + // Pass-through only (see runAppServerTurn): the review caller supplies the + // watchdog default via resolveReviewTurnIdleTimeoutMs(). + const turnIdleTimeoutMs = options.turnIdleTimeoutMs; + const inferredCompletionQuietMs = options.inferredCompletionQuietMs; + const sandbox = options.sandbox ?? "read-only"; + + return withAppServer(cwd, async (client) => { + emitProgress(options.onProgress, "Starting Codex investigation thread.", "starting"); + const startResponse = await startThread(client, cwd, { + model: options.model, + sandbox, + ephemeral: true, + threadName: null + }); + const threadId = startResponse.thread.id; + emitProgress(options.onProgress, `Thread ready (${threadId}).`, "starting", { threadId }); + + let turnCount = 0; + let truncated = false; + let totalCommandsRun = 0; + const aggregatedCommandExecutions = []; + const aggregatedFileChanges = []; + + for (let i = 1; i <= maxInvestigationTurns; i += 1) { + const promptText = i === 1 ? investigatePrompt : INVESTIGATION_CONTINUATION_CUE; + emitProgress(options.onProgress, `Investigation turn ${i}.`, "investigating"); + + let turnState; + try { + turnState = await captureTurn( + client, + threadId, + () => + client.request("turn/start", { + threadId, + input: buildTurnInput(promptText), + model: options.model ?? null, + effort: options.effort ?? null, + outputSchema: null + }), + { onProgress: options.onProgress, turnIdleTimeoutMs, inferredCompletionQuietMs } + ); + } catch (transportError) { + return { + status: 1, + threadId, + turnId: null, + finalMessage: "", + reasoningSummary: [], + turn: null, + error: { message: transportError?.message ?? String(transportError) }, + stderr: cleanCodexStderr(client.stderr), + fileChanges: aggregatedFileChanges, + touchedFiles: collectTouchedFiles(aggregatedFileChanges), + commandExecutions: aggregatedCommandExecutions, + investigation: { turnCount, truncated: false } + }; + } + + turnCount = i; + const turnCommandCount = turnState.commandExecutions.length; + totalCommandsRun += turnCommandCount; + for (const cmd of turnState.commandExecutions) { + aggregatedCommandExecutions.push(cmd); + } + for (const change of turnState.fileChanges) { + aggregatedFileChanges.push(change); + } + + // The app-server multiplexes transient retry notices (e.g. + // "Reconnecting... 1/5") onto the same `error` notification channel as + // fatal turn failures, and the capture state records the last one seen + // without clearing it. A reconnect that recovers still drives the turn + // to turn/completed with an agent message, so treating any `error` as a + // hard abort would skip the schema-enforced finalize turn and hand the + // raw investigation prose to the JSON parser. Only abort when the turn + // produced no usable output — i.e. it did not recover. + const turnHadAgentMessage = Boolean(turnState.lastAgentMessage); + const turnRecovered = turnHadAgentMessage && turnState.finalTurn?.status === "completed"; + + if (turnState.error && !turnRecovered) { + return { + status: buildResultStatus(turnState), + threadId, + turnId: turnState.turnId, + finalMessage: turnState.lastAgentMessage, + reasoningSummary: turnState.reasoningSummary, + turn: turnState.finalTurn, + error: turnState.error, + stderr: cleanCodexStderr(client.stderr), + fileChanges: aggregatedFileChanges, + touchedFiles: collectTouchedFiles(aggregatedFileChanges), + commandExecutions: aggregatedCommandExecutions, + investigation: { turnCount, truncated: false } + }; + } + + // Convergence: a turn that produces no commands and emits an agent + // message is the contract the investigate prompt teaches the model + // ("a summary message with no further command calls signals readiness"). + // The legacy check required `phase: "final_answer"`, but recon turns + // run with outputSchema=null so the app-server does not always tag + // messages with that phase — leading to runaway turns where the model + // keeps insisting it has converged but the loop refuses to listen. + const converged = turnCommandCount === 0 && turnHadAgentMessage; + if (converged) { + break; + } + + if (i === maxInvestigationTurns) { + truncated = true; + } + } + + if (totalCommandsRun === 0) { + truncated = true; + } + + emitProgress(options.onProgress, "Investigation complete; finalizing structured output.", "finalizing"); + + // The finalize turn is supposed to emit only the structured JSON. In + // practice the model sometimes emits a tool-call stub instead (e.g. + // {"cmd": "wc -l ..."}) — if any commands ran during finalize, treat + // that as a contract violation and retry once with a sharper prompt. + const STRICT_FINALIZE_REMINDER = + "STRICT FINALIZE: do not run any shell commands. Output ONLY the JSON " + + "matching the schema, with no prose, no tool calls, and nothing else.\n\n"; + let finalizeState; + let finalizeAttempts = 0; + const MAX_FINALIZE_ATTEMPTS = 2; + while (finalizeAttempts < MAX_FINALIZE_ATTEMPTS) { + finalizeAttempts += 1; + const promptText = finalizeAttempts === 1 + ? finalizePrompt + : STRICT_FINALIZE_REMINDER + finalizePrompt; + try { + finalizeState = await captureTurn( + client, + threadId, + () => + client.request("turn/start", { + threadId, + input: buildTurnInput(promptText), + model: options.model ?? null, + effort: resolveFinalizeEffort(options.effort ?? null), + outputSchema: options.outputSchema ?? null + }), + { onProgress: options.onProgress, turnIdleTimeoutMs, inferredCompletionQuietMs } + ); + } catch (transportError) { + return { + status: 1, + threadId, + turnId: null, + finalMessage: "", + reasoningSummary: [], + turn: null, + error: { message: transportError?.message ?? String(transportError) }, + stderr: cleanCodexStderr(client.stderr), + fileChanges: aggregatedFileChanges, + touchedFiles: collectTouchedFiles(aggregatedFileChanges), + commandExecutions: aggregatedCommandExecutions, + investigation: { turnCount, truncated } + }; + } + + // If the finalize turn ran commands, the model violated the contract. + // Retry once with a stricter prompt; if it still fails, accept the + // (likely-malformed) output and let the parser surface the error. + if (finalizeState.commandExecutions.length === 0) { + break; + } + if (finalizeAttempts < MAX_FINALIZE_ATTEMPTS) { + emitProgress( + options.onProgress, + "Finalize turn ran commands; retrying with stricter prompt.", + "finalizing" + ); + // Aggregate the wasted commands from this (about-to-be-superseded) + // attempt so the caller still sees what happened. The final attempt's + // executions are aggregated once after the loop, so only fold in + // attempts we are leaving behind here — otherwise the last attempt + // would be counted twice. + for (const cmd of finalizeState.commandExecutions) { + aggregatedCommandExecutions.push(cmd); + } + for (const change of finalizeState.fileChanges) { + aggregatedFileChanges.push(change); + } + } + } + + for (const cmd of finalizeState.commandExecutions) { + aggregatedCommandExecutions.push(cmd); + } + for (const change of finalizeState.fileChanges) { + aggregatedFileChanges.push(change); + } + + return { + status: buildResultStatus(finalizeState), + threadId, + turnId: finalizeState.turnId, + finalMessage: finalizeState.lastAgentMessage, + reasoningSummary: finalizeState.reasoningSummary, + turn: finalizeState.finalTurn, + error: finalizeState.error, + stderr: cleanCodexStderr(client.stderr), + fileChanges: aggregatedFileChanges, + touchedFiles: collectTouchedFiles(aggregatedFileChanges), + commandExecutions: aggregatedCommandExecutions, + investigation: { turnCount, truncated } + }; + }); +} + export async function findLatestTaskThread(cwd) { const availability = getCodexAvailability(cwd); if (!availability.available) { diff --git a/plugins/codex/scripts/lib/git.mjs b/plugins/codex/scripts/lib/git.mjs index c401f0ca1..7ec36a5a9 100644 --- a/plugins/codex/scripts/lib/git.mjs +++ b/plugins/codex/scripts/lib/git.mjs @@ -5,8 +5,18 @@ import { isProbablyText } from "./fs.mjs"; import { formatCommandFailure, runCommand, runCommandChecked } from "./process.mjs"; const MAX_UNTRACKED_BYTES = 24 * 1024; -const DEFAULT_INLINE_DIFF_MAX_FILES = 2; +// Inline-diff embeds full file contents into the prompt and pins outputSchema +// on a single turn — there is no recovery if the model wants to investigate +// before producing the verdict. Keep this path narrow: only single-file +// reviews of small diffs use it. Anything larger falls through to the +// two-phase self-collect path which can tolerate exploratory turns. +const DEFAULT_INLINE_DIFF_MAX_FILES = 1; const DEFAULT_INLINE_DIFF_MAX_BYTES = 256 * 1024; +// Multi-turn investigation can tolerate a much larger inline payload than the +// single-shot path: the model reads the diff as evidence and still has +// follow-up turns for anything it needs beyond it. 1MB is a safe share of a +// 272K-token context window. +const DEFAULT_INVESTIGATION_INLINE_MAX_BYTES = 1024 * 1024; // Git is directly executable on Windows. Repository-derived arguments must never pass through a shell. function git(cwd, args, options = {}) { @@ -37,6 +47,15 @@ function normalizeMaxInlineDiffBytes(value) { return Math.floor(parsed); } +function normalizeInvestigationInlineMaxBytes(value) { + const raw = value ?? process.env.CODEX_COMPANION_INVESTIGATION_INLINE_MAX_BYTES; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) { + return DEFAULT_INVESTIGATION_INLINE_MAX_BYTES; + } + return Math.floor(parsed); +} + function measureGitOutputBytes(cwd, args, maxBytes) { const result = git(cwd, args, { maxBuffer: maxBytes + 1 }); if (result.error && /** @type {NodeJS.ErrnoException} */ (result.error).code === "ENOBUFS") { @@ -194,32 +213,52 @@ function formatSection(title, body) { return [`## ${title}`, "", body.trim() ? body.trim() : "(none)", ""].join("\n"); } -function formatUntrackedFile(cwd, relativePath) { +// Single source of truth for whether an untracked file's contents can be +// embedded into the inline prompt. Returns either a `skipped` reason (the file +// is a directory, too large, binary, or unreadable) or the file `content`. +function classifyUntrackedFile(cwd, relativePath) { const absolutePath = path.join(cwd, relativePath); let stat; try { stat = fs.statSync(absolutePath); } catch { - return `### ${relativePath}\n(skipped: broken symlink or unreadable file)`; + return { skipped: "broken symlink or unreadable file" }; } if (stat.isDirectory()) { - return `### ${relativePath}\n(skipped: directory)`; + return { skipped: "directory" }; } if (stat.size > MAX_UNTRACKED_BYTES) { - return `### ${relativePath}\n(skipped: ${stat.size} bytes exceeds ${MAX_UNTRACKED_BYTES} byte limit)`; + return { skipped: `${stat.size} bytes exceeds ${MAX_UNTRACKED_BYTES} byte limit` }; } let buffer; try { buffer = fs.readFileSync(absolutePath); } catch { - return `### ${relativePath}\n(skipped: broken symlink or unreadable file)`; + return { skipped: "broken symlink or unreadable file" }; } if (!isProbablyText(buffer)) { - return `### ${relativePath}\n(skipped: binary file)`; + return { skipped: "binary file" }; } - return [`### ${relativePath}`, "```", buffer.toString("utf8").trimEnd(), "```"].join("\n"); + return { content: buffer.toString("utf8").trimEnd() }; +} + +// True when at least one untracked file's contents cannot be embedded inline. +// An untracked file never appears in `git diff`, so a single skipped untracked +// file otherwise looks like a 1-file, 0-byte diff and slips onto the inline +// path — where the prompt embeds only a `(skipped: ...)` marker and forbids +// shell, leaving the reviewer nothing to inspect. +function hasSkippedUntrackedContent(cwd, untracked) { + return untracked.some((file) => Boolean(classifyUntrackedFile(cwd, file).skipped)); +} + +function formatUntrackedFile(cwd, relativePath) { + const classified = classifyUntrackedFile(cwd, relativePath); + if (classified.skipped) { + return `### ${relativePath}\n(skipped: ${classified.skipped})`; + } + return [`### ${relativePath}`, "```", classified.content, "```"].join("\n"); } function collectWorkingTreeContext(cwd, state, options = {}) { @@ -294,6 +333,18 @@ function buildAdversarialCollectionGuidance(options = {}) { return "Use the repository context below as primary evidence."; } + if (options.investigationInline) { + const fed = + "The full diff is embedded below as primary evidence — do not re-derive it with git commands. Run read-only commands only when you need context beyond the diff itself: surrounding code, callers, history, or tests."; + // Untracked files never appear in `git diff`, and oversized/binary ones are + // reduced to a `(skipped: ...)` marker. Without this clause the fed wording + // would claim complete evidence while telling the model not to go looking. + if (options.hasSkippedUntracked) { + return `${fed} Some untracked files could not be embedded — read them directly with read-only commands.`; + } + return fed; + } + return "The repository context below is a lightweight summary. Inspect the target diff yourself with read-only git commands before finalizing findings."; } @@ -302,35 +353,69 @@ export function collectReviewContext(cwd, target, options = {}) { const currentBranch = getCurrentBranch(repoRoot); const maxInlineFiles = normalizeMaxInlineFiles(options.maxInlineFiles); const maxInlineDiffBytes = normalizeMaxInlineDiffBytes(options.maxInlineDiffBytes); + const investigationInlineMaxBytes = normalizeInvestigationInlineMaxBytes(options.investigationInlineMaxBytes); + // Measure up to whichever budget is larger so a diff that overflows the + // single-shot cap still yields a real byte count for the investigation check. + const measureCap = Math.max(maxInlineDiffBytes, investigationInlineMaxBytes); let details; - let includeDiff; + // singleShotInline decides inline-diff vs self-collect routing; investigationInline + // decides whether the self-collect prompt carries the diff. Keep them distinct. + let singleShotInline; + let investigationInline; let diffBytes; + // Only meaningful in working-tree mode; a branch diff ignores the working tree. + let fedDiffOmitsUntracked = false; if (target.mode === "working-tree") { const state = getWorkingTreeState(repoRoot); + // hasSkippedUntrackedContent() stats and reads every untracked file, and two + // decisions below consult it. Memoize so it runs at most once, and keep it + // lazy so neither decision pays for it when a cheaper conjunct already lost. + let skippedUntracked = null; + const hasSkippedUntracked = () => { + if (skippedUntracked === null) { + skippedUntracked = hasSkippedUntrackedContent(repoRoot, state.untracked); + } + return skippedUntracked; + }; diffBytes = measureCombinedGitOutputBytes( repoRoot, [ ["diff", "--cached", "--binary", "--no-ext-diff", "--submodule=diff"], ["diff", "--binary", "--no-ext-diff", "--submodule=diff"] ], - maxInlineDiffBytes + measureCap ); - includeDiff = + singleShotInline = options.includeDiff ?? (listUniqueFiles(state.staged, state.unstaged, state.untracked).length <= maxInlineFiles && - diffBytes <= maxInlineDiffBytes); - details = collectWorkingTreeContext(repoRoot, state, { includeDiff }); + diffBytes <= maxInlineDiffBytes && + !hasSkippedUntracked()); + // Only the byte bound matters here: skipped untracked content is fine + // because the multi-turn path still has read-only shell to inspect it. + investigationInline = + options.includeDiff === undefined && !singleShotInline && diffBytes <= investigationInlineMaxBytes; + // The fed diff is then incomplete, so the guidance must say so rather than + // claim the embedded diff is the whole change. + fedDiffOmitsUntracked = investigationInline && hasSkippedUntracked(); + details = collectWorkingTreeContext(repoRoot, state, { + includeDiff: singleShotInline || investigationInline + }); } else { const comparison = buildBranchComparison(repoRoot, target.baseRef); const fileCount = gitChecked(repoRoot, ["diff", "--name-only", comparison.commitRange]).stdout.trim().split("\n").filter(Boolean).length; diffBytes = measureGitOutputBytes( repoRoot, ["diff", "--binary", "--no-ext-diff", "--submodule=diff", comparison.commitRange], - maxInlineDiffBytes + measureCap ); - includeDiff = options.includeDiff ?? (fileCount <= maxInlineFiles && diffBytes <= maxInlineDiffBytes); - details = collectBranchContext(repoRoot, target.baseRef, { includeDiff, comparison }); + singleShotInline = options.includeDiff ?? (fileCount <= maxInlineFiles && diffBytes <= maxInlineDiffBytes); + investigationInline = + options.includeDiff === undefined && !singleShotInline && diffBytes <= investigationInlineMaxBytes; + details = collectBranchContext(repoRoot, target.baseRef, { + includeDiff: singleShotInline || investigationInline, + comparison + }); } return { @@ -340,8 +425,13 @@ export function collectReviewContext(cwd, target, options = {}) { target, fileCount: details.changedFiles.length, diffBytes, - inputMode: includeDiff ? "inline-diff" : "self-collect", - collectionGuidance: buildAdversarialCollectionGuidance({ includeDiff }), + inputMode: singleShotInline ? "inline-diff" : "self-collect", + investigationInline, + collectionGuidance: buildAdversarialCollectionGuidance({ + includeDiff: singleShotInline, + investigationInline, + hasSkippedUntracked: fedDiffOmitsUntracked + }), ...details }; } diff --git a/plugins/codex/scripts/lib/render.mjs b/plugins/codex/scripts/lib/render.mjs index 2ec185236..e27764b4c 100644 --- a/plugins/codex/scripts/lib/render.mjs +++ b/plugins/codex/scripts/lib/render.mjs @@ -208,7 +208,39 @@ export function renderSetupReport(report) { return `${lines.join("\n").trimEnd()}\n`; } +function appendInvestigationBanner(lines, meta) { + if (meta.investigation?.truncated === true) { + const turns = meta.investigation.turnCount ?? "?"; + lines.push( + `Investigation truncated at ${turns} turns; findings may be shallow. Use --max-investigation-turns to raise the cap.`, + "" + ); + } +} + export function renderReviewResult(parsedResult, meta) { + if (parsedResult.failed) { + // The run failed before producing a structured verdict (transport drop, + // idle timeout, soft turn error). Report the real reason rather than + // parsing the leftover prose as JSON and surfacing a misleading parse error. + const lines = [ + `# Codex ${meta.reviewLabel}`, + "", + "Codex could not complete the review.", + "", + `- Reason: ${parsedResult.failureMessage ?? "Codex run failed before producing output."}` + ]; + appendInvestigationBanner(lines, meta); + + if (parsedResult.rawOutput) { + lines.push("", "Partial investigation output:", "", "```text", parsedResult.rawOutput, "```"); + } + + appendReasoningSection(lines, meta.reasoningSummary ?? parsedResult.reasoningSummary); + + return `${lines.join("\n").trimEnd()}\n`; + } + if (!parsedResult.parsed) { const lines = [ `# Codex ${meta.reviewLabel}`, @@ -217,6 +249,7 @@ export function renderReviewResult(parsedResult, meta) { "", `- Parse error: ${parsedResult.parseError}` ]; + appendInvestigationBanner(lines, meta); if (parsedResult.rawOutput) { lines.push("", "Raw final message:", "", "```text", parsedResult.rawOutput, "```"); @@ -237,6 +270,7 @@ export function renderReviewResult(parsedResult, meta) { "", `- Validation error: ${validationError}` ]; + appendInvestigationBanner(lines, meta); if (parsedResult.rawOutput) { lines.push("", "Raw final message:", "", "```text", parsedResult.rawOutput, "```"); @@ -258,6 +292,7 @@ export function renderReviewResult(parsedResult, meta) { data.summary, "" ]; + appendInvestigationBanner(lines, meta); if (findings.length === 0) { lines.push("No material findings."); diff --git a/tests/commands.test.mjs b/tests/commands.test.mjs index c34b06059..5d2d07e4a 100644 --- a/tests/commands.test.mjs +++ b/tests/commands.test.mjs @@ -49,7 +49,9 @@ test("adversarial review command uses AskUserQuestion and background Bash while assert.match(source, /```bash/); assert.match(source, /```typescript/); assert.match(source, /adversarial-review "\$ARGUMENTS"/); - assert.match(source, /\[--scope auto\|working-tree\|branch\] \[focus \.\.\.\]/); + assert.match(source, /\[--scope auto\|working-tree\|branch\]/); + assert.match(source, /\[--max-investigation-turns N\]/); + assert.match(source, /\[focus \.\.\.\]/); assert.match(source, /run_in_background:\s*true/); assert.match(source, /command:\s*`node "\$\{CLAUDE_PLUGIN_ROOT\}\/scripts\/codex-companion\.mjs" adversarial-review "\$ARGUMENTS"`/); assert.match(source, /description:\s*"Codex adversarial review"/); diff --git a/tests/fake-codex-fixture.mjs b/tests/fake-codex-fixture.mjs index f83c96a0d..bf1b8adcc 100644 --- a/tests/fake-codex-fixture.mjs +++ b/tests/fake-codex-fixture.mjs @@ -2,7 +2,7 @@ import fs from "node:fs"; import path from "node:path"; import process from "node:process"; -import { writeExecutable } from "./helpers.mjs"; +import { makeTempDir, writeExecutable } from "./helpers.mjs"; export function installFakeCodex(binDir, behavior = "review-ok") { const statePath = path.join(binDir, "fake-codex-state.json"); @@ -16,6 +16,7 @@ const readline = require("node:readline"); const STATE_PATH = ${JSON.stringify(statePath)}; const BEHAVIOR = ${JSON.stringify(behavior)}; const interruptibleTurns = new Map(); + let serializedBusyThread = null; function loadState() { if (!fs.existsSync(STATE_PATH)) { @@ -234,7 +235,7 @@ function structuredReviewPayload(prompt) { function taskPayload(prompt, resume) { if (prompt.includes("") && prompt.includes("Only review the work from the previous Claude turn.")) { - if (BEHAVIOR === "adversarial-clean") { + if (BEHAVIOR === "adversarial-clean" || BEHAVIOR === "gate-recovered") { return "ALLOW: No blocking issues found in the previous turn."; } return "BLOCK: Missing empty-state guard in src/app.js:4-6."; @@ -413,31 +414,176 @@ rl.on("line", (line) => { } const turnId = nextTurnId(state); send({ id: message.id, result: { turn: buildTurn(turnId), reviewThreadId: reviewThread.id } }); - emitTurnCompleted(reviewThread.id, turnId, [ - { - started: { type: "enteredReviewMode", id: turnId, review: "current changes" } - }, - ...(BEHAVIOR === "with-reasoning" - ? [ - { - completed: { - type: "reasoning", - id: "reasoning_" + turnId, - summary: [{ text: "Reviewed the changed files and checked the likely regression paths." }], - content: [] - } - } - ] - : []), - { - completed: { type: "exitedReviewMode", id: turnId, review: nativeReviewText(message.params.target) } - } - ]); + + // Queue-driven mode lets a test script the review text and inject a + // transient (recovered) error to exercise the recovered-status path. + const reviewEntry = + BEHAVIOR === "queue-driven" && state.queue && state.queue.length > 0 + ? state.queue.shift() + : null; + if (reviewEntry) { + saveState(state); + } + const reviewText = reviewEntry && typeof reviewEntry.reviewText === "string" + ? reviewEntry.reviewText + : nativeReviewText(message.params.target); + + send({ method: "turn/started", params: { threadId: reviewThread.id, turn: buildTurn(turnId) } }); + send({ + method: "item/started", + params: { threadId: reviewThread.id, turnId, item: { type: "enteredReviewMode", id: turnId, review: "current changes" } } + }); + if (BEHAVIOR === "with-reasoning") { + send({ + method: "item/completed", + params: { + threadId: reviewThread.id, + turnId, + item: { + type: "reasoning", + id: "reasoning_" + turnId, + summary: [{ text: "Reviewed the changed files and checked the likely regression paths." }], + content: [] + } + } + }); + } + send({ + method: "item/completed", + params: { threadId: reviewThread.id, turnId, item: { type: "exitedReviewMode", id: turnId, review: reviewText } } + }); + if (reviewEntry && reviewEntry.turnError) { + send({ method: "error", params: { threadId: reviewThread.id, turnId, error: { message: reviewEntry.turnError.message } } }); + } + send({ method: "turn/completed", params: { threadId: reviewThread.id, turn: buildTurn(turnId, "completed") } }); break; } case "turn/start": { const thread = ensureThread(state, message.params.threadId); + + if (BEHAVIOR === "queue-driven") { + if (!state.requests) { state.requests = []; } + state.requests.push({ method: "turn/start", params: message.params }); + + if (state.serialize) { + if (serializedBusyThread === thread.id) { + // A turn is already open on this thread. The real app-server queues + // this turn/start and (in the bug) never opens it: no result, no + // turn/started, no turn/completed. Persist the recorded request, + // then hang. + saveState(state); + break; + } + // Only the normal completion paths below (delayCompletedMs / the + // synchronous turn/completed) clear serializedBusyThread. Do NOT + // combine serialize with hang/error entries (cueThenHang, + // hangNoResponse, hangAfterStarted, foreignChatterThenHang, + // rpcError) when a SUBSEQUENT queued turn is expected to open — those + // branches break early and leave the thread marked busy on purpose. + serializedBusyThread = thread.id; + } + + const turnId = nextTurnId(state); + thread.updatedAt = now(); + + const entry = (state.queue && state.queue.length > 0) ? state.queue.shift() : null; + saveState(state); + + if (entry && entry.rpcError) { + send({ id: message.id, error: { code: -32000, message: entry.rpcError.message } }); + break; + } + + if (entry && entry.hangNoResponse) { + // Model a half-dead upstream: the request is received but the + // server never replies (no result, no turn/started, no + // turn/completed). The client-side turn/start promise stays + // pending forever -- this is the real "stuck at turn N" signature. + break; + } + + if (entry && entry.hangAfterStarted) { + // Announce the turn so the client buffers a turn/started carrying the + // id (populating pendingTurnId), but never send the turn/start RPC + // result and never complete the turn. Models a delayed RPC reply on a + // half-dead link, exercising Defect C. + send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } }); + break; + } + + send({ id: message.id, result: { turn: buildTurn(turnId) } }); + send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } }); + + if (entry && entry.foreignChatterThenHang) { + const { count = 5, everyMs = 50 } = entry.foreignChatterThenHang; + const foreignThreadId = thread.id + "-foreign"; + const foreignTurnId = turnId + "-foreign"; + // Foreign-thread traffic: must NOT re-arm our turn's watchdog. + for (let n = 0; n < count; n += 1) { + setTimeout(() => { + send({ + method: "item/completed", + params: { + threadId: foreignThreadId, + turnId: foreignTurnId, + item: { type: "agentMessage", id: "foreign_" + n, text: "noise", phase: "analysis" } + } + }); + }, everyMs * (n + 1)); + } + // Never emit turn/completed for OUR turn -> the watchdog must fire. + break; + } + + const commands = (entry && entry.commands) || []; + let cmdCounter = 0; + for (const cmd of commands) { + const itemId = "cmd_" + turnId + "_" + (cmdCounter++); + send({ method: "item/started", params: { threadId: thread.id, turnId, item: { type: "commandExecution", id: itemId, command: cmd.command, status: "in_progress" } } }); + send({ method: "item/completed", params: { threadId: thread.id, turnId, item: { type: "commandExecution", id: itemId, command: cmd.command, exitCode: cmd.exitCode ?? 0, status: "completed" } } }); + } + + if (entry && entry.finalAnswer) { + const phase = entry.finalAnswer.phase ?? "final_answer"; + send({ method: "item/completed", params: { threadId: thread.id, turnId, item: { type: "agentMessage", id: "msg_" + turnId, text: entry.finalAnswer.text, phase } } }); + } + + if (entry && entry.lateFinalAnswer) { + const lateTurnId = turnId; + setTimeout(() => { + send({ method: "item/completed", params: { threadId: thread.id, turnId: lateTurnId, item: { type: "agentMessage", id: "late_" + lateTurnId, text: entry.lateFinalAnswer.text, phase: "final_answer" } } }); + }, entry.lateFinalAnswer.afterMs ?? 100); + } + + if (entry && entry.cueThenHang) { + // Emit only the readiness cue (already sent above); never send a real + // turn/completed. Exercises the Defect A gate: a plain turn must not + // infer completion from the cue. + break; + } + + if (entry && entry.turnError) { + send({ method: "error", params: { threadId: thread.id, turnId, error: { message: entry.turnError.message } } }); + } + + if (!entry) { + send({ method: "item/completed", params: { threadId: thread.id, turnId, item: { type: "agentMessage", id: "msg_" + turnId, text: "", phase: "agent_message" } } }); + } + + if (entry && entry.delayCompletedMs) { + const completedTurnId = turnId; + setTimeout(() => { + if (state.serialize) { serializedBusyThread = null; } + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(completedTurnId, "completed") } }); + }, entry.delayCompletedMs); + } else { + if (state.serialize) { serializedBusyThread = null; } + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } }); + } + break; + } + const prompt = (message.params.input || []) .filter((item) => item.type === "text") .map((item) => item.text) @@ -602,6 +748,19 @@ rl.on("line", (line) => { interruptibleTurns.set(turnId, { threadId: thread.id, timer }); } else if (BEHAVIOR === "slow-task") { emitTurnCompletedLater(thread.id, turnId, items, 400); + } else if (BEHAVIOR === "gate-recovered") { + // Recovered transient: emit the agent message, then a stale "error" + // notice, then turn/completed. The turn still has usable output, so the + // companion must exit 0 (resolveRunExitStatus) and the gate must parse + // the ALLOW answer rather than block on a phantom failure. + send({ method: "turn/started", params: { threadId: thread.id, turn: buildTurn(turnId) } }); + for (const entry of items) { + if (entry && entry.completed) { + send({ method: "item/completed", params: { threadId: thread.id, turnId, item: entry.completed } }); + } + } + send({ method: "error", params: { threadId: thread.id, turnId, error: { message: "Reconnecting... 1/5" } } }); + send({ method: "turn/completed", params: { threadId: thread.id, turn: buildTurn(turnId, "completed") } }); } else { emitTurnCompleted(thread.id, turnId, items); } @@ -656,3 +815,91 @@ export function buildEnv(binDir) { PATH: `${binDir}${sep}${process.env.PATH}` }; } + +/** + * Sets up a queue-driven fake Codex harness for multi-turn tests. + * Returns a handle with helpers for scripting turn responses and + * inspecting captured requests. + */ +export function setupFakeCodex({ cwd } = {}) { + const binDir = makeTempDir("codex-queue-driven-"); + installFakeCodex(binDir, "queue-driven"); + + const statePath = path.join(binDir, "fake-codex-state.json"); + const initialState = { + nextThreadId: 1, + nextTurnId: 1, + appServerStarts: 0, + threads: [], + capabilities: null, + lastInterrupt: null, + queue: [], + requests: [], + serialize: false + }; + fs.writeFileSync(statePath, JSON.stringify(initialState, null, 2)); + + const sep = process.platform === "win32" ? ";" : ":"; + process.env.PATH = `${binDir}${sep}${process.env.PATH}`; + + const env = buildEnv(binDir); + const resolvedCwd = cwd || process.cwd(); + + function readState() { + return JSON.parse(fs.readFileSync(statePath, "utf8")); + } + + function writeState(state) { + fs.writeFileSync(statePath, JSON.stringify(state, null, 2)); + } + + return { + cwd: resolvedCwd, + env, + binDir, + queueTurnResponse(entry) { + const state = readState(); + if (!state.queue) { state.queue = []; } + state.queue.push(entry); + writeState(state); + }, + queueTurnRpcError({ message }) { + const state = readState(); + if (!state.queue) { state.queue = []; } + state.queue.push({ rpcError: { message } }); + writeState(state); + }, + queueTurnHang() { + // The server receives the turn/start but never responds, modelling a + // half-dead upstream connection. Used to exercise the idle timeout. + const state = readState(); + if (!state.queue) { state.queue = []; } + state.queue.push({ hangNoResponse: true }); + writeState(state); + }, + queueTurnHangAfterStarted() { + const state = readState(); + if (!state.queue) { state.queue = []; } + state.queue.push({ hangAfterStarted: true }); + writeState(state); + }, + enableSerialization() { + const state = readState(); + state.serialize = true; + writeState(state); + }, + // `requests` re-reads the state file each access; assign to a local variable for repeated use. + get requests() { + const state = readState(); + return state.requests ?? []; + }, + close() { + const sep = process.platform === "win32" ? ";" : ":"; + process.env.PATH = (process.env.PATH ?? "") + .split(sep) + .filter((entry) => entry !== binDir) + .join(sep); + fs.rmSync(binDir, { recursive: true, force: true }); + } + }; +} diff --git a/tests/fake-codex-fixture.test.mjs b/tests/fake-codex-fixture.test.mjs new file mode 100644 index 000000000..3dcaeb5ba --- /dev/null +++ b/tests/fake-codex-fixture.test.mjs @@ -0,0 +1,203 @@ +import test from "node:test"; +import assert from "node:assert/strict"; + +import { setupFakeCodex } from "./fake-codex-fixture.mjs"; +import { resolveReviewTurnIdleTimeoutMs, resolveRunExitStatus, runAppServerTurn } from "../plugins/codex/scripts/lib/codex.mjs"; +import { makeTempDir } from "./helpers.mjs"; + +test("resolveReviewTurnIdleTimeoutMs defaults the review watchdog and honors explicit values", () => { + // The idle watchdog is a REVIEW concern (a stalled review should not hang + // forever). It must NOT be baked into the shared runAppServerTurn default, + // because /codex:task also calls runAppServerTurn and a long-thinking task + // would then be aborted at this threshold with no task-level knob. The default + // lives in this review-only helper instead. + assert.equal(resolveReviewTurnIdleTimeoutMs(undefined), 1_200_000, "review default is 1200s"); + assert.equal(resolveReviewTurnIdleTimeoutMs(null), 1_200_000, "null falls back to the review default"); + assert.equal(resolveReviewTurnIdleTimeoutMs(300), 300, "explicit ms passes through"); + assert.equal(resolveReviewTurnIdleTimeoutMs(0), 1_200_000, "zero/invalid falls back to the review default"); +}); + +test("runAppServerTurn passes through an absent idle timeout (task path arms no watchdog)", async () => { + // Regression guard for the task path: executeTaskRun calls runAppServerTurn + // without turnIdleTimeoutMs. runAppServerTurn must NOT inject the review + // default — it passes the absent value straight to captureTurn, which arms no + // watchdog for undefined. A normal turn therefore completes cleanly with no + // idle-timeout error. (The review default lives in resolveReviewTurnIdleTimeoutMs; + // the explicit-timeout abort is covered by the inline-review hang test below.) + const cwd = makeTempDir("codex-queue-test-"); + const handle = setupFakeCodex({ cwd }); + try { + handle.queueTurnResponse({ commands: [], finalAnswer: { text: "task done" } }); + + const result = await runAppServerTurn(cwd, { prompt: "long-running task, no watchdog" }); + + assert.equal(result.finalMessage, "task done", "task turn should complete normally with no timeout"); + assert.equal(result.error ?? null, null, "no idle-timeout error when no watchdog is configured"); + } finally { + handle.close(); + } +}); + +test("queue-driven fake: final answer is returned via runAppServerTurn", async () => { + const cwd = makeTempDir("codex-queue-test-"); + const handle = setupFakeCodex({ cwd }); + try { + handle.queueTurnResponse({ finalAnswer: { text: "hi" } }); + + const result = await runAppServerTurn(cwd, { + prompt: "say hi" + }); + + assert.equal(result.finalMessage, "hi"); + assert.equal(result.status, 0); + } finally { + handle.close(); + } +}); + +test("queue-driven fake: requests are captured with params", async () => { + const cwd = makeTempDir("codex-queue-test-"); + const handle = setupFakeCodex({ cwd }); + try { + handle.queueTurnResponse({ finalAnswer: { text: "captured" } }); + + await runAppServerTurn(cwd, { + prompt: "check capture" + }); + + const turnStarts = handle.requests.filter((r) => r.method === "turn/start"); + assert.equal(turnStarts.length, 1); + // Verify the captured params include the input text + const inputTexts = turnStarts[0].params.input + .filter((item) => item.type === "text") + .map((item) => item.text); + assert.ok(inputTexts.some((text) => text.includes("check capture"))); + } finally { + handle.close(); + } +}); + +test("queue-driven fake: commandExecution items are emitted and captured", async () => { + const cwd = makeTempDir("codex-queue-test-"); + const handle = setupFakeCodex({ cwd }); + try { + handle.queueTurnResponse({ + commands: [{ command: "git diff", exitCode: 0 }], + finalAnswer: { text: "done" } + }); + + const result = await runAppServerTurn(cwd, { + prompt: "run commands" + }); + + assert.equal(result.finalMessage, "done"); + assert.equal(result.commandExecutions.length, 1); + assert.equal(result.commandExecutions[0].command, "git diff"); + assert.equal(result.commandExecutions[0].exitCode, 0); + } finally { + handle.close(); + } +}); + +test("queue-driven fake: RPC error causes runAppServerTurn to reject", async () => { + const cwd = makeTempDir("codex-queue-test-"); + const handle = setupFakeCodex({ cwd }); + try { + handle.queueTurnRpcError({ message: "boom" }); + + await assert.rejects( + runAppServerTurn(cwd, { prompt: "should fail" }), + (error) => { + assert.ok(error.message.includes("boom")); + return true; + } + ); + } finally { + handle.close(); + } +}); + +test("queue-driven fake: inline turn that never responds is aborted by the idle timeout", async () => { + // The inline (single-turn) review path runs through runAppServerTurn. A + // half-dead upstream that accepts turn/start but never responds would hang + // the RPC forever without a watchdog. The advertised --turn-idle-timeout must + // apply here too, not only on the self-collect path. + const cwd = makeTempDir("codex-queue-test-"); + const handle = setupFakeCodex({ cwd }); + try { + handle.queueTurnHang(); + + const start = Date.now(); + await assert.rejects( + runAppServerTurn(cwd, { prompt: "should time out", turnIdleTimeoutMs: 300 }), + (error) => { + assert.match(error.message, /idle|timed out|timeout/i, "error should explain the idle timeout"); + return true; + } + ); + const elapsed = Date.now() - start; + assert.ok(elapsed < 15000, `must abort promptly, not hang (took ${elapsed}ms)`); + } finally { + handle.close(); + } +}); + +test("queue-driven fake: soft error (turnError) is captured", async () => { + const cwd = makeTempDir("codex-queue-test-"); + const handle = setupFakeCodex({ cwd }); + try { + handle.queueTurnResponse({ + finalAnswer: { text: "partial" }, + turnError: { message: "soft failure" } + }); + + const result = await runAppServerTurn(cwd, { + prompt: "trigger soft error" + }); + + // The turn still completes, but state.error is set + assert.equal(result.finalMessage, "partial"); + assert.ok(result.error); + assert.equal(result.error.message, "soft failure"); + } finally { + handle.close(); + } +}); + +test("resolveRunExitStatus treats a completed turn with usable text as success despite a stale error", () => { + // Recovered transient: turn completed, has usable text, but result.status is 1 + // (buildResultStatus saw the stale `error`). Must resolve to 0. + assert.equal( + resolveRunExitStatus({ turn: { status: "completed" }, status: 1 }, "ALLOW: looks fine"), + 0, + "completed turn with usable text overrides the stale non-zero status" + ); + + // Genuine failure: no usable text => keep the raw status. + assert.equal( + resolveRunExitStatus({ turn: { status: "completed" }, status: 1 }, " "), + 1, + "completed turn with no usable text keeps the failure status" + ); + + // Genuine failure: turn did not complete => keep the raw status even with text. + assert.equal( + resolveRunExitStatus({ turn: { status: "failed" }, status: 1 }, "some text"), + 1, + "non-completed turn keeps the failure status" + ); + + // Clean success: status already 0 => stays 0. + assert.equal( + resolveRunExitStatus({ turn: { status: "completed" }, status: 0 }, "done"), + 0, + "a clean success stays 0" + ); + + // Missing turn => not recovered => raw status. + assert.equal( + resolveRunExitStatus({ turn: null, status: 1 }, "text"), + 1, + "absent turn cannot be recovered; keep the raw status" + ); +}); diff --git a/tests/git.test.mjs b/tests/git.test.mjs index 5b5c266ee..78b7b7478 100644 --- a/tests/git.test.mjs +++ b/tests/git.test.mjs @@ -115,6 +115,28 @@ test("collectReviewContext keeps inline diffs for tiny adversarial reviews", () assert.match(context.content, /INLINE_MARKER/); }); +test("collectReviewContext routes 2-file changes to self-collect (inline cap is 1)", () => { + // Regression guard: a 2-file change used to slip into inline-diff because + // the cap was 2, embedding both files into a single-turn schema-pinned + // prompt — and the model often responded with a tool-call stub instead + // of the review JSON. Two files now go through the two-phase self-collect + // path which tolerates exploratory turns. + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "seed.js"), "export const value = 'seed';\n"); + run("git", ["add", "seed.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + fs.writeFileSync(path.join(cwd, "doc-one.md"), "# planning doc\n".repeat(50)); + fs.writeFileSync(path.join(cwd, "doc-two.md"), "# spec doc\n".repeat(50)); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(context.fileCount, 2); + assert.equal(context.inputMode, "self-collect", + "2-file changes must NOT be inlined; they hit the schema-pinned single-turn bug otherwise"); +}); + test("collectReviewContext skips untracked directories in working tree review", () => { const cwd = makeTempDir(); initGitRepo(cwd); @@ -148,7 +170,10 @@ test("collectReviewContext skips broken untracked symlinks instead of crashing", assert.match(context.content, /skipped: broken symlink or unreadable file/i); }); -test("collectReviewContext falls back to lightweight context for larger adversarial reviews", () => { +test("collectReviewContext routes larger adversarial reviews to self-collect with the diff fed", () => { + // Routing guard: >1 changed file must never take the single-shot inline path. + // The diff itself is still embedded — it is well under the investigation + // budget, and re-deriving it would cost the reviewer an expensive first turn. const cwd = makeTempDir(); initGitRepo(cwd); for (const name of ["a.js", "b.js", "c.js"]) { @@ -165,13 +190,15 @@ test("collectReviewContext falls back to lightweight context for larger adversar assert.equal(context.inputMode, "self-collect"); assert.equal(context.fileCount, 3); - assert.match(context.collectionGuidance, /lightweight summary/i); - assert.match(context.collectionGuidance, /read-only git commands/i); - assert.doesNotMatch(context.content, /SELF_COLLECT_MARKER_[ABC]/); - assert.match(context.content, /## Changed Files/); + assert.equal(context.investigationInline, true); + assert.match(context.collectionGuidance, /full diff is embedded below/i); + assert.match(context.content, /SELF_COLLECT_MARKER_A/); }); -test("collectReviewContext falls back to lightweight context for oversized single-file diffs", () => { +test("collectReviewContext falls back to self-collect for oversized single-file diffs", () => { + // The two budgets are independent: exceeding the single-shot byte cap moves + // the review off the inline-diff path, but the diff still fits the (much + // larger) investigation budget, so it is fed to the multi-turn prompt. const cwd = makeTempDir(); initGitRepo(cwd); fs.writeFileSync(path.join(cwd, "app.js"), "export const value = 'v1';\n"); @@ -185,11 +212,14 @@ test("collectReviewContext falls back to lightweight context for oversized singl assert.equal(context.fileCount, 1); assert.equal(context.inputMode, "self-collect"); assert.ok(context.diffBytes > 128); - assert.doesNotMatch(context.content, /xxx/); - assert.match(context.content, /## Changed Files/); + assert.equal(context.investigationInline, true); + assert.match(context.content, /xxx/); }); test("collectReviewContext keeps untracked file content in lightweight working tree context", () => { + // Blind working-tree path (diff over the investigation budget): untracked + // files never appear in `git diff`, so their contents must still be embedded + // or the summary hides them entirely. const cwd = makeTempDir(); initGitRepo(cwd); for (const name of ["a.js", "b.js"]) { @@ -202,11 +232,243 @@ test("collectReviewContext keeps untracked file content in lightweight working t fs.writeFileSync(path.join(cwd, "new-risk.js"), 'export const value = "UNTRACKED_RISK_MARKER";\n'); const target = resolveReviewTarget(cwd, {}); - const context = collectReviewContext(cwd, target); + const context = collectReviewContext(cwd, target, { investigationInlineMaxBytes: 10 }); assert.equal(context.inputMode, "self-collect"); assert.equal(context.fileCount, 3); + assert.equal(context.investigationInline, false); assert.doesNotMatch(context.content, /TRACKED_MARKER_[AB]/); assert.match(context.content, /## Untracked Files/); assert.match(context.content, /UNTRACKED_RISK_MARKER/); }); + +test("collectReviewContext routes a single oversized untracked file to self-collect", () => { + // An untracked file never shows up in `git diff`, so its size does not count + // toward diffBytes. A single untracked file >24 KiB therefore looked like a + // 1-file, 0-byte diff and slipped onto the inline path — where the prompt + // embeds only a `(skipped: ...)` marker AND forbids shell. The reviewer could + // then only approve/guess. Skipped untracked content must fall through to + // self-collect so Codex can read the file with read-only commands. + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "seed.js"), "export const value = 'seed';\n"); + run("git", ["add", "seed.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + // One untracked file, contents exceed MAX_UNTRACKED_BYTES (24 KiB). + fs.writeFileSync(path.join(cwd, "big-untracked.txt"), "x".repeat(30 * 1024)); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(context.fileCount, 1); + assert.equal(context.inputMode, "self-collect", + "a skipped untracked file must NOT be inlined; the prompt would embed only a (skipped) marker while forbidding shell"); +}); + +test("collectReviewContext routes a single binary untracked file to self-collect", () => { + // Same hazard as the oversized case: a small binary untracked file is within + // the byte/file caps but its contents are skipped as `(skipped: binary file)`, + // so the inline prompt would show nothing useful while forbidding shell. + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "seed.js"), "export const value = 'seed';\n"); + run("git", ["add", "seed.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + // Untracked binary file: NUL bytes make isProbablyText() false. + fs.writeFileSync(path.join(cwd, "blob.bin"), Buffer.from([0, 1, 2, 0, 3, 4, 0])); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(context.fileCount, 1); + assert.equal(context.inputMode, "self-collect", + "a binary untracked file must NOT be inlined; its contents are skipped in the embedded prompt"); +}); + +test("collectReviewContext still inlines a single small text untracked file", () => { + // Guard the fix from over-reaching: an untracked file whose contents ARE + // embeddable (small, text) must stay on the inline path. + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "seed.js"), "export const value = 'seed';\n"); + run("git", ["add", "seed.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + fs.writeFileSync(path.join(cwd, "small-new.js"), "export const v = 'INLINE_UNTRACKED_MARKER';\n"); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(context.fileCount, 1); + assert.equal(context.inputMode, "inline-diff"); + assert.match(context.content, /INLINE_UNTRACKED_MARKER/); +}); + +test("mid-size branch diff self-collects WITH the full diff embedded (investigation inline)", () => { + // The multi-turn path used to be handed a stat-only summary, so the reviewer + // burned its first (very expensive) reasoning turns re-deriving the diff with + // `git diff`. Anything under the investigation budget is now fed inline while + // still routing to the multi-turn path. + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "seed.js"), "export const value = 'seed';\n"); + run("git", ["add", "seed.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + run("git", ["checkout", "-b", "feature/test"], { cwd }); + fs.writeFileSync(path.join(cwd, "doc-one.md"), "# planning doc\n".repeat(50)); + fs.writeFileSync(path.join(cwd, "doc-two.md"), "# spec doc\n".repeat(50)); + run("git", ["add", "doc-one.md", "doc-two.md"], { cwd }); + run("git", ["commit", "-m", "docs"], { cwd }); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(context.fileCount, 2); + assert.equal(context.inputMode, "self-collect"); + assert.equal(context.investigationInline, true); + assert.match(context.content, /## Branch Diff/); + assert.match(context.content, /diff --git/); + assert.match(context.collectionGuidance, /full diff is embedded below/i); +}); + +test("diff above the investigation budget self-collects blind (lightweight summary)", () => { + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "seed.js"), "export const value = 'seed';\n"); + run("git", ["add", "seed.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + run("git", ["checkout", "-b", "feature/test"], { cwd }); + fs.writeFileSync(path.join(cwd, "doc-one.md"), "# planning doc\n".repeat(50)); + fs.writeFileSync(path.join(cwd, "doc-two.md"), "# spec doc\n".repeat(50)); + run("git", ["add", "doc-one.md", "doc-two.md"], { cwd }); + run("git", ["commit", "-m", "docs"], { cwd }); + + const target = resolveReviewTarget(cwd, {}); + process.env.CODEX_COMPANION_INVESTIGATION_INLINE_MAX_BYTES = "10"; + try { + const context = collectReviewContext(cwd, target); + assert.equal(context.inputMode, "self-collect"); + assert.equal(context.investigationInline, false); + assert.match(context.content, /## Changed Files/); + assert.doesNotMatch(context.content, /diff --git/); + assert.match(context.collectionGuidance, /lightweight summary/i); + assert.match(context.collectionGuidance, /read-only git commands/i); + } finally { + delete process.env.CODEX_COMPANION_INVESTIGATION_INLINE_MAX_BYTES; + } +}); + +test("tiny single-file diff still routes to inline-diff (single-shot path unchanged)", () => { + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "app.js"), "console.log('v1');\n"); + run("git", ["add", "app.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + fs.writeFileSync(path.join(cwd, "app.js"), "console.log('INLINE_MARKER');\n"); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(context.inputMode, "inline-diff"); + assert.equal(context.investigationInline, false); + assert.match(context.collectionGuidance, /primary evidence/i); +}); + +test("explicit includeDiff:false still forces blind self-collect", () => { + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "app.js"), "console.log('v1');\n"); + run("git", ["add", "app.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + fs.writeFileSync(path.join(cwd, "app.js"), "console.log('OVERRIDE_MARKER');\n"); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target, { includeDiff: false }); + + assert.equal(context.inputMode, "self-collect"); + assert.equal(context.investigationInline, false); + assert.match(context.collectionGuidance, /lightweight summary/i); + assert.doesNotMatch(context.content, /OVERRIDE_MARKER/); +}); + +test("mid-size working-tree diff also gets investigation inline", () => { + const cwd = makeTempDir(); + initGitRepo(cwd); + for (const name of ["a.js", "b.js"]) { + fs.writeFileSync(path.join(cwd, name), `export const value = "${name}-v1";\n`); + } + run("git", ["add", "a.js", "b.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + fs.writeFileSync(path.join(cwd, "a.js"), 'export const value = "STAGED_FED_MARKER";\n'); + run("git", ["add", "a.js"], { cwd }); + fs.writeFileSync(path.join(cwd, "b.js"), 'export const value = "UNSTAGED_FED_MARKER";\n'); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(target.mode, "working-tree"); + assert.equal(context.fileCount, 2); + assert.equal(context.inputMode, "self-collect"); + assert.equal(context.investigationInline, true); + assert.match(context.content, /## Staged Diff/); + assert.match(context.content, /## Unstaged Diff/); + assert.match(context.content, /diff --git/); + assert.match(context.content, /STAGED_FED_MARKER/); + assert.match(context.content, /UNSTAGED_FED_MARKER/); + assert.match(context.collectionGuidance, /full diff is embedded below/i); + // Nothing was skipped, so the guidance must NOT hedge about untracked files. + assert.doesNotMatch(context.collectionGuidance, /could not be embedded/i); +}); + +test("fed working-tree guidance warns when untracked content could not be embedded", () => { + // An untracked file never appears in `git diff`, and oversized/binary ones are + // reduced to a `(skipped: ...)` marker. Telling the model "the full diff is + // embedded — do not re-derive it" would then assert complete evidence while + // discouraging the one action that recovers the missing content: reading the + // file. The fed wording must own that gap. + const cwd = makeTempDir(); + initGitRepo(cwd); + for (const name of ["a.js", "b.js"]) { + fs.writeFileSync(path.join(cwd, name), `export const value = "${name}-v1";\n`); + } + run("git", ["add", "a.js", "b.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + fs.writeFileSync(path.join(cwd, "a.js"), 'export const value = "SKIPPED_CASE_MARKER_A";\n'); + fs.writeFileSync(path.join(cwd, "b.js"), 'export const value = "SKIPPED_CASE_MARKER_B";\n'); + // Untracked and over MAX_UNTRACKED_BYTES (24 KiB), so its contents are skipped. + fs.writeFileSync(path.join(cwd, "big-untracked.txt"), "x".repeat(30 * 1024)); + + const target = resolveReviewTarget(cwd, {}); + const context = collectReviewContext(cwd, target); + + assert.equal(target.mode, "working-tree"); + assert.equal(context.inputMode, "self-collect"); + assert.equal(context.investigationInline, true, "routing is unchanged: this still takes the fed path"); + assert.match(context.content, /SKIPPED_CASE_MARKER_A/); + assert.match(context.content, /skipped: 30720 bytes/); + assert.match(context.collectionGuidance, /full diff is embedded below/i); + assert.match(context.collectionGuidance, /read them directly with read-only commands/i); +}); + +test("fed branch-mode guidance keeps the unqualified wording (no untracked concept)", () => { + const cwd = makeTempDir(); + initGitRepo(cwd); + fs.writeFileSync(path.join(cwd, "seed.js"), "export const value = 'seed';\n"); + run("git", ["add", "seed.js"], { cwd }); + run("git", ["commit", "-m", "init"], { cwd }); + run("git", ["checkout", "-b", "feature/test"], { cwd }); + fs.writeFileSync(path.join(cwd, "doc-one.md"), "# planning doc\n".repeat(50)); + fs.writeFileSync(path.join(cwd, "doc-two.md"), "# spec doc\n".repeat(50)); + run("git", ["add", "doc-one.md", "doc-two.md"], { cwd }); + run("git", ["commit", "-m", "docs"], { cwd }); + // An untracked file that WOULD be skipped in working-tree mode. A branch diff + // does not consider the working tree at all, so the wording must not hedge. + fs.writeFileSync(path.join(cwd, "big-untracked.txt"), "x".repeat(30 * 1024)); + + const target = resolveReviewTarget(cwd, { base: "main" }); + const context = collectReviewContext(cwd, target); + + assert.equal(target.mode, "branch"); + assert.equal(context.investigationInline, true); + assert.match(context.collectionGuidance, /full diff is embedded below/i); + assert.doesNotMatch(context.collectionGuidance, /could not be embedded/i); +}); diff --git a/tests/investigation.test.mjs b/tests/investigation.test.mjs new file mode 100644 index 000000000..bf9720818 --- /dev/null +++ b/tests/investigation.test.mjs @@ -0,0 +1,1532 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; + +import { setupFakeCodex } from "./fake-codex-fixture.mjs"; +import { runAppServerInvestigation } from "../plugins/codex/scripts/lib/codex.mjs"; +import { makeTempDir } from "./helpers.mjs"; + +// Structured JSON payloads used by multiple tests. +const STRUCTURED_REVIEW = JSON.stringify({ + verdict: "needs-attention", + summary: "Concern X.", + findings: [{ + severity: "high", + title: "Race", + file: "a.js", + line_start: 10, + line_end: 12, + confidence: 0.8, + body: "Potential race condition.", + recommendation: "Add a mutex." + }], + next_steps: [] +}); + +const APPROVE_REVIEW = JSON.stringify({ + verdict: "approve", + summary: "No material issues found.", + findings: [], + next_steps: [] +}); + +test("converges when Codex emits a final-answer turn with no commands", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands, no final answer + fake.queueTurnResponse({ + commands: [{ command: "git diff HEAD~1", exitCode: 0 }], + finalAnswer: null + }); + // Recon turn 2: no commands, final answer => converges + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Investigation done." } + }); + // Finalize turn 3 + fake.queueTurnResponse({ + finalAnswer: { text: STRUCTURED_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate the changes.", + finalizePrompt: "Produce your structured verdict.", + outputSchema: { type: "object", required: ["verdict"] } + }); + + assert.equal(result.investigation.turnCount, 2); + assert.equal(result.investigation.truncated, false); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3, "should have 3 turn/start requests (2 recon + 1 finalize)"); + } finally { + fake.close(); + } +}); + +test("converges when agentMessage has no `final_answer` phase tag (real-world case)", async () => { + // In production, recon turns run with outputSchema=null, and the + // app-server does NOT always tag agent messages with phase="final_answer". + // The convergence detector must treat any 0-command turn that emits an + // agent message as convergence — otherwise the model keeps insisting it + // has converged but the loop refuses to stop. + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + commands: [{ command: "git diff HEAD~1", exitCode: 0 }], + finalAnswer: null + }); + // Recon turn 2: no commands, agent message WITHOUT final_answer phase + // => must still converge (this is what real codex sends in recon mode). + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "My investigation is complete.", phase: "agent_message" } + }); + fake.queueTurnResponse({ + finalAnswer: { text: STRUCTURED_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] } + }); + + assert.equal(result.investigation.turnCount, 2, + "convergence on the no-command + agentMessage turn must not be missed"); + assert.equal(result.investigation.truncated, false); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3, "2 recon + 1 finalize"); + } finally { + fake.close(); + } +}); + +test("respects maxInvestigationTurns and marks truncated", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Queue 3 recon turns: all have commands, no final answer. + // (With maxInvestigationTurns=3 the loop will exhaust here.) + for (let i = 0; i < 3; i++) { + fake.queueTurnResponse({ + commands: [{ command: `check-${i}`, exitCode: 0 }] + }); + } + // Finalize turn: pure JSON, zero commands. + fake.queueTurnResponse({ + finalAnswer: { text: APPROVE_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + maxInvestigationTurns: 3 + }); + + assert.equal(result.investigation.turnCount, 3); + assert.equal(result.investigation.truncated, true); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 4, "3 recon + 1 finalize"); + } finally { + fake.close(); + } +}); + +test("turn with both finalAnswer and commands does not converge", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands AND final answer => does NOT converge + fake.queueTurnResponse({ + commands: [{ command: "grep -r TODO", exitCode: 0 }], + finalAnswer: { text: "Partial finding." } + }); + // Recon turn 2: only final answer, no commands => converges + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Investigation done." } + }); + // Finalize turn + fake.queueTurnResponse({ + finalAnswer: { text: APPROVE_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize." + }); + + assert.equal(result.investigation.turnCount, 2); + assert.equal(result.investigation.truncated, false); + } finally { + fake.close(); + } +}); + +test("outputSchema is null on recon turns and set on finalize turn", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands + fake.queueTurnResponse({ + commands: [{ command: "cat file.js", exitCode: 0 }] + }); + // Recon turn 2: final answer, no commands => converges + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Done investigating." } + }); + // Finalize turn + fake.queueTurnResponse({ + finalAnswer: { text: STRUCTURED_REVIEW } + }); + + const schema = { type: "object", required: ["verdict"] }; + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: schema + }); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3); + + // Recon turns must have outputSchema === null + assert.equal(starts[0].params.outputSchema, null, "recon turn 1 outputSchema should be null"); + assert.equal(starts[1].params.outputSchema, null, "recon turn 2 outputSchema should be null"); + + // Finalize turn must have the schema + assert.deepEqual(starts[2].params.outputSchema, schema, "finalize turn should have the outputSchema"); + } finally { + fake.close(); + } +}); + +test("phase-1 soft error (turn/failed) aborts before phase-2 finalize", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands, normal + fake.queueTurnResponse({ + commands: [{ command: "git log --oneline", exitCode: 0 }] + }); + // Recon turn 2: soft error + fake.queueTurnResponse({ + turnError: { message: "model produced unrenderable response" } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize." + }); + + assert.ok(result.error, "result should have an error"); + assert.match(result.error.message, /model produced unrenderable response/); + assert.equal(result.investigation.turnCount, 2, "soft-error turn IS counted"); + // A turn/completed with status="completed" can still arrive after an + // error notification, so result.status must be derived from the error, + // not from finalTurn.status. CI/automation relies on this. + assert.equal(result.status, 1, "soft-error path returns numeric status 1"); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 2, "NO finalize turn should be attempted"); + } finally { + fake.close(); + } +}); + +test("phase-1 hard error (transport throw) aborts before phase-2 finalize", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands, normal + fake.queueTurnResponse({ + commands: [{ command: "git status", exitCode: 0 }] + }); + // Recon turn 2: RPC error (transport throw) + fake.queueTurnRpcError({ message: "ECONNRESET" }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize." + }); + + assert.ok(result.error, "result should have an error"); + assert.match(result.error.message, /ECONNRESET/); + assert.equal(result.investigation.turnCount, 1, "hard error returns BEFORE incrementing turnCount"); + assert.equal(result.status, 1, "transport-error path returns numeric status"); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 2, "the failing turn was still attempted"); + } finally { + fake.close(); + } +}); + +test("turn that never responds is aborted by the idle timeout (no infinite hang)", async () => { + // Production hang: turn 1 runs fine, then the next turn/start is sent but the + // half-dead upstream never responds — no `turn/started`, no completion. The + // RPC promise has no timeout, so the loop would await forever (observed as + // "stuck at Investigation turn 2"). An idle timeout must abort it gracefully. + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: makes progress, does NOT converge (has commands). + fake.queueTurnResponse({ + commands: [{ command: "git diff", exitCode: 0 }], + finalAnswer: null + }); + // Recon turn 2: the server receives turn/start but never responds. + fake.queueTurnHang(); + + const start = Date.now(); + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + turnIdleTimeoutMs: 300 + }); + const elapsed = Date.now() - start; + + assert.ok(result.error, "a timed-out run must carry an error"); + assert.match(result.error.message, /idle|timed out|timeout/i, "error should explain the idle timeout"); + assert.equal(result.status, 1, "idle-timeout path returns non-zero status"); + assert.ok(elapsed < 15000, `must abort promptly, not hang (took ${elapsed}ms)`); + } finally { + fake.close(); + } +}); + +test("a turn that keeps emitting progress is NOT killed by the idle timeout", async () => { + // Regression guard: the idle timer must reset on every progress notification, + // so a healthy turn running many commands (longer than the idle window in + // aggregate, but never silent for that long) must not be aborted. + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Turn 1: several commands, then converges on turn 2 with a summary. + fake.queueTurnResponse({ + commands: [ + { command: "c1", exitCode: 0 }, + { command: "c2", exitCode: 0 }, + { command: "c3", exitCode: 0 } + ], + finalAnswer: null + }); + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Done." } }); + fake.queueTurnResponse({ finalAnswer: { text: STRUCTURED_REVIEW } }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + turnIdleTimeoutMs: 300 + }); + + assert.equal(result.error ?? null, null, "a healthy turn must not be timed out"); + assert.equal(result.finalMessage, STRUCTURED_REVIEW, "should reach finalize normally"); + } finally { + fake.close(); + } +}); + +test("transient reconnect during recon recovers and still reaches finalize", async () => { + // The app-server multiplexes transient retry notices ("Reconnecting... N/5") + // onto the same `error` notification channel as fatal turn failures. A + // reconnect that recovers still drives the turn to turn/completed with an + // agent message. The loop must NOT treat that as a fatal abort — doing so + // skips the schema-enforced finalize turn and hands the raw investigation + // prose to the JSON parser (observed: `Unexpected token 'I', "Investigat"...`). + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: ran commands, hit a transient reconnect, still produced a + // summary agent message and completed (i.e. it recovered). + fake.queueTurnResponse({ + commands: [{ command: "git diff", exitCode: 0 }], + finalAnswer: { text: "Investigation complete. I'm ready for finalization." }, + turnError: { message: "Reconnecting... 1/5" } + }); + // Recon turn 2: 0 commands + agent message => converges. + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Confirmed, ready." } + }); + // Finalize turn 3: structured JSON. + fake.queueTurnResponse({ + finalAnswer: { text: STRUCTURED_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] } + }); + + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3, "should run 2 recon + 1 finalize (finalize NOT skipped)"); + assert.equal(result.finalMessage, STRUCTURED_REVIEW, + "finalMessage should be the structured JSON, not the investigation prose"); + } finally { + fake.close(); + } +}); + +test("finalize turn that runs commands triggers one strict-prompt retry", async () => { + // Production observation: the model occasionally emits a tool-call stub + // during finalize (e.g. {"cmd": "wc -l ..."}) instead of the structured + // JSON. When that happens the finalize turn has commandExecutions.length > 0. + // The orchestrator should detect this contract violation and retry once + // with a stricter prompt. + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon: converge. + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Investigation done." } + }); + // First finalize attempt: model misbehaves and runs a command. + fake.queueTurnResponse({ + commands: [{ command: "wc -l README.md", exitCode: 0 }], + finalAnswer: { text: "{\"cmd\":\"wc -l README.md\"}" } + }); + // Second finalize attempt: model behaves and emits proper JSON. + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: APPROVE_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] } + }); + + assert.equal(result.finalMessage, APPROVE_REVIEW, + "the second (well-behaved) finalize attempt should be the final message"); + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3, "1 recon + 2 finalize attempts"); + + // The retry must use a stricter prompt distinct from the first try. + const finalizeStarts = starts.slice(1); // first start is recon + assert.match( + finalizeStarts[1].params.input?.[0]?.text ?? "", + /STRICT FINALIZE/, + "retry prompt must include the stricter directive" + ); + assert.doesNotMatch( + finalizeStarts[0].params.input?.[0]?.text ?? "", + /STRICT FINALIZE/, + "first finalize attempt uses the normal prompt" + ); + } finally { + fake.close(); + } +}); + +test("finalize retry gives up after the second attempt and surfaces the output", async () => { + // If the model misbehaves twice in a row, the orchestrator must not loop + // forever — it should accept the second attempt's output and let the + // upstream parser produce a useful validation error. + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Investigation done." } + }); + // Both finalize attempts misbehave. + fake.queueTurnResponse({ + commands: [{ command: "wc -l a", exitCode: 0 }], + finalAnswer: { text: "{\"cmd\":\"wc -l a\"}" } + }); + fake.queueTurnResponse({ + commands: [{ command: "wc -l b", exitCode: 0 }], + finalAnswer: { text: "{\"cmd\":\"wc -l b\"}" } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] } + }); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3, "1 recon + 2 finalize attempts only — no infinite loop"); + assert.equal(result.finalMessage, "{\"cmd\":\"wc -l b\"}", + "the second-attempt output is surfaced so the parser can flag it"); + + // The two finalize attempts ran one command each; neither should be + // double-counted. (Regression: the final attempt was previously appended + // both in-loop and after the loop, duplicating its command/file traces.) + const cmds = result.commandExecutions.map((c) => c.command); + assert.deepEqual(cmds, ["wc -l a", "wc -l b"], + "each finalize attempt's commands recorded exactly once, no duplicates"); + } finally { + fake.close(); + } +}); + +test("phase-2 finalize transport error preserves investigation metadata", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands + fake.queueTurnResponse({ + commands: [{ command: "git diff", exitCode: 0 }] + }); + // Recon turn 2: converge with final answer + no commands + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Investigation done." } + }); + // Finalize turn: RPC error + fake.queueTurnRpcError({ message: "ETIMEDOUT during finalize" }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize." + }); + + assert.ok(result.error, "result should carry the finalize error"); + assert.match(result.error.message, /ETIMEDOUT during finalize/); + assert.equal(result.investigation.turnCount, 2, "investigation completed both recon turns before finalize failed"); + assert.equal(result.investigation.truncated, false, "investigation converged; not truncated"); + assert.equal(result.status, 1, "finalize-error path returns numeric status"); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3, "2 recon + 1 finalize attempted"); + } finally { + fake.close(); + } +}); + +test("converges with zero commands flags truncated=true", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: only final answer, no commands => converges immediately + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Nothing to investigate." } + }); + // Finalize turn + fake.queueTurnResponse({ + finalAnswer: { text: APPROVE_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize." + }); + + assert.equal(result.investigation.turnCount, 1); + assert.equal(result.investigation.truncated, true, "zero commands across investigation => truncated"); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 2, "1 recon + 1 finalize"); + } finally { + fake.close(); + } +}); + +test("recon turn 1 sends the investigate prompt; turn 2+ sends the continuation cue", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands + fake.queueTurnResponse({ + commands: [{ command: "ls", exitCode: 0 }] + }); + // Recon turn 2: commands + fake.queueTurnResponse({ + commands: [{ command: "cat a.js", exitCode: 0 }] + }); + // Recon turn 3: final answer, no commands => converges + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "All done." } + }); + // Finalize turn + fake.queueTurnResponse({ + finalAnswer: { text: APPROVE_REVIEW } + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "FULL INVESTIGATE PROMPT: look at the code", + finalizePrompt: "FINALIZE PROMPT: produce verdict" + }); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 4, "3 recon + 1 finalize"); + + // Extract input text from each turn/start + const inputTexts = starts.map((s) => + (s.params.input || []) + .filter((p) => p.type === "text") + .map((p) => p.text) + .join("") + ); + + // Turn 1: investigate prompt + assert.match(inputTexts[0], /FULL INVESTIGATE PROMPT/, "turn 1 should use investigate prompt"); + // Turn 2 and 3: continuation cue + assert.equal(inputTexts[1], "Continue your investigation.", "turn 2 should use continuation cue"); + assert.equal(inputTexts[2], "Continue your investigation.", "turn 3 should use continuation cue"); + // Turn 4: finalize prompt + assert.match(inputTexts[3], /FINALIZE PROMPT/, "turn 4 should use finalize prompt"); + } finally { + fake.close(); + } +}); + +// ------------------------------------------------------------------- +// Integration tests: subprocess-based end-to-end companion tests +// ------------------------------------------------------------------- + +import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { spawnSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +const COMPANION_PATH = fileURLToPath( + new URL("../plugins/codex/scripts/codex-companion.mjs", import.meta.url) +); + +function makeSelfCollectGitFixture() { + // 3+ changed files triggers self-collect (DEFAULT_INLINE_DIFF_MAX_FILES = 2) + const root = mkdtempSync(path.join(tmpdir(), "codex-self-collect-test-")); + spawnSync("git", ["init", "-q", "-b", "main"], { cwd: root }); + spawnSync("git", ["config", "user.email", "test@example.com"], { cwd: root }); + spawnSync("git", ["config", "user.name", "Test"], { cwd: root }); + spawnSync("git", ["config", "commit.gpgsign", "false"], { cwd: root }); + writeFileSync(path.join(root, "README.md"), "# repo\n"); + spawnSync("git", ["add", "."], { cwd: root }); + spawnSync("git", ["commit", "-q", "-m", "init"], { cwd: root }); + spawnSync("git", ["checkout", "-q", "-b", "feature"], { cwd: root }); + mkdirSync(path.join(root, "src"), { recursive: true }); + for (let i = 0; i < 5; i += 1) { + writeFileSync(path.join(root, "src", `f${i}.js`), `export const v${i} = ${i};\n`); + } + spawnSync("git", ["add", "."], { cwd: root }); + spawnSync("git", ["commit", "-q", "-m", "feature"], { cwd: root }); + return root; +} + +function makeInlineGitFixture() { + // 1 changed file stays on inline-diff path + const root = mkdtempSync(path.join(tmpdir(), "codex-inline-test-")); + spawnSync("git", ["init", "-q", "-b", "main"], { cwd: root }); + spawnSync("git", ["config", "user.email", "test@example.com"], { cwd: root }); + spawnSync("git", ["config", "user.name", "Test"], { cwd: root }); + spawnSync("git", ["config", "commit.gpgsign", "false"], { cwd: root }); + writeFileSync(path.join(root, "README.md"), "# repo\n"); + spawnSync("git", ["add", "."], { cwd: root }); + spawnSync("git", ["commit", "-q", "-m", "init"], { cwd: root }); + spawnSync("git", ["checkout", "-q", "-b", "feature"], { cwd: root }); + writeFileSync(path.join(root, "one.js"), "export const v = 1;\n"); + spawnSync("git", ["add", "."], { cwd: root }); + spawnSync("git", ["commit", "-q", "-m", "tiny"], { cwd: root }); + return root; +} + +function makeCleanDefaultBranchGitFixture() { + // A clean working tree sitting ON the default branch (main). resolveReviewTarget + // falls back to a branch diff against the detected default (main) — but HEAD IS + // main, so merge-base == HEAD and the diff is empty. Nothing to review. + const root = mkdtempSync(path.join(tmpdir(), "codex-empty-diff-test-")); + spawnSync("git", ["init", "-q", "-b", "main"], { cwd: root }); + spawnSync("git", ["config", "user.email", "test@example.com"], { cwd: root }); + spawnSync("git", ["config", "user.name", "Test"], { cwd: root }); + spawnSync("git", ["config", "commit.gpgsign", "false"], { cwd: root }); + writeFileSync(path.join(root, "README.md"), "# repo\n"); + spawnSync("git", ["add", "."], { cwd: root }); + spawnSync("git", ["commit", "-q", "-m", "init"], { cwd: root }); + return root; +} + +function runCompanion(args, env) { + return spawnSync("node", [COMPANION_PATH, ...args], { + env: { ...process.env, ...env }, + encoding: "utf8", + timeout: 30000 + }); +} + +test("self-collect path uses runAppServerInvestigation end-to-end", async () => { + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: runs a command, no convergence + fake.queueTurnResponse({ + commands: [{ command: "git diff main...HEAD", exitCode: 0 }], + finalAnswer: null + }); + // Recon turn 2: no commands, final answer => converges + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Investigation done." } + }); + // Finalize turn: structured output + fake.queueTurnResponse({ + finalAnswer: { + text: JSON.stringify({ + verdict: "needs-attention", + summary: "Found risk in src/f1.js.", + findings: [{ + severity: "high", + title: "Unguarded export", + file: "src/f1.js", + line_start: 1, + line_end: 1, + confidence: 0.7, + body: "Module exports v1 with no validation.", + recommendation: "Add validation." + }], + next_steps: [] + }) + } + }); + + const result = runCompanion( + ["adversarial-review", "--base", "main", "--scope", "branch", "--cwd", cwd, "--json"], + fake.env + ); + + assert.equal(result.status, 0, `expected exit 0, stderr: ${result.stderr}`); + + // stdout may contain progress lines followed by JSON; parse the last JSON object + const stdout = result.stdout.trim(); + const payload = JSON.parse(stdout); + assert.ok(payload.investigation, "self-collect payload must have investigation field"); + assert.equal(payload.investigation.turnCount, 2); + assert.equal(payload.investigation.truncated, false); + assert.equal(payload.result?.verdict, "needs-attention"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("inline run that completes with no message renders as no-content, not a fake parse error (e2e)", async () => { + // Reproduces the production failure (thread 019ea22a): a status-0 turn that + // emitted only reasoning and no agent message. finalMessage was "", but the + // run did NOT error — so the old code skipped the `failed` branch, ran the + // empty string through parseStructuredOutput's `!rawOutput` path, and + // produced an EMPTY parseError. The user saw "Codex did not return valid + // structured JSON / - Parse error:" with nothing after the colon. An empty + // completed run is a no-content result, NOT a malformed-JSON parse error. + const cwd = makeInlineGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + // Single inline turn that completes with NO finalAnswer => empty message, + // status 0, no error (the fake emits turn/completed with no agentMessage). + fake.queueTurnResponse({ commands: [], finalAnswer: null }); + + const result = runCompanion( + ["adversarial-review", "--base", "main", "--scope", "branch", "--cwd", cwd, "--json"], + fake.env + ); + + const payload = JSON.parse(result.stdout.trim()); + assert.equal(payload.failed, true, "an empty completed run must be flagged failed"); + assert.equal(payload.parseError, null, "must NOT fabricate a JSON parse error for empty output"); + assert.match( + payload.failureMessage ?? "", + /no review content|no final message|returned no/i, + "failure reason must explain the empty output honestly" + ); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("clean repo on the default branch short-circuits without calling the model (e2e)", async () => { + // Reproduces the empty-diff trigger: running adversarial-review on a clean + // working tree while sitting ON the default branch makes the branch + // comparison resolve merge-base == HEAD, i.e. an empty diff. The old code + // fed that empty diff to the model, which burned reasoning tokens and + // returned nothing. With nothing to review there is no reason to call the + // model at all — the run should short-circuit to an approve verdict. + const cwd = makeCleanDefaultBranchGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + // Queue a turn so that IF the model is (wrongly) called, we can detect it. + fake.queueTurnResponse({ + finalAnswer: { text: JSON.stringify({ verdict: "approve", summary: "x", findings: [], next_steps: [] }) } + }); + + const result = runCompanion( + ["adversarial-review", "--cwd", cwd, "--json"], + fake.env + ); + + assert.equal(result.status, 0, `expected exit 0, stderr: ${result.stderr}`); + const payload = JSON.parse(result.stdout.trim()); + assert.equal(payload.result?.verdict, "approve", "an empty diff should short-circuit to approve"); + + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 0, "no model turn should be started when there is nothing to review"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("inline-diff path does not call runAppServerInvestigation", async () => { + const cwd = makeInlineGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + // Single turn: inline path uses runAppServerTurn + fake.queueTurnResponse({ + finalAnswer: { + text: JSON.stringify({ + verdict: "approve", + summary: "No material issues found.", + findings: [], + next_steps: [] + }) + } + }); + + const result = runCompanion( + ["adversarial-review", "--base", "main", "--scope", "branch", "--cwd", cwd, "--json"], + fake.env + ); + + assert.equal(result.status, 0, `expected exit 0, stderr: ${result.stderr}`); + const payload = JSON.parse(result.stdout.trim()); + assert.equal(payload.investigation, undefined, + "inline-path payload must not carry the investigation field"); + + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 1, "inline path is single-turn"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("self-collect run failure renders as failure, not a fake JSON parse error (e2e)", async () => { + // Reproduces the production failure: the connection to the upstream drops + // mid-investigation (the `turn/start` request rejects, modelling a closed + // socket). The companion must report the transport failure reason, NOT + // JSON.parse the leftover prose and surface a misleading parse error. + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnRpcError({ + message: "stream disconnected before completion: error sending request" + }); + + const result = runCompanion( + ["adversarial-review", "--base", "main", "--scope", "branch", "--cwd", cwd, "--json"], + fake.env + ); + + assert.notEqual(result.status, 0, "a failed run should exit non-zero"); + const payload = JSON.parse(result.stdout.trim()); + assert.equal(payload.failed, true, "payload should be flagged failed"); + assert.match(payload.failureMessage ?? "", /stream disconnected/, "failure reason preserved"); + assert.equal(payload.parseError, null, "must NOT produce a JSON parse error for a transport failure"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("finalize turn that recovered from a transient reconnect keeps its valid verdict (e2e)", async () => { + // Regression guard for the finalize boundary: a transient reconnect during + // the finalize turn sets `error` (so status becomes 1), but the turn still + // emitted valid structured JSON. The companion must render that verdict, NOT + // discard it as "could not complete the review" — the mirror of fix #1. + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands, no converge. + fake.queueTurnResponse({ commands: [{ command: "git diff main...HEAD", exitCode: 0 }], finalAnswer: null }); + // Recon turn 2: converges. + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + // Finalize turn: valid JSON AND a transient reconnect error (recovered). + fake.queueTurnResponse({ + finalAnswer: { + text: JSON.stringify({ + verdict: "needs-attention", + summary: "Found risk in src/f1.js.", + findings: [{ + severity: "high", title: "Unguarded export", file: "src/f1.js", + line_start: 1, line_end: 1, confidence: 0.7, + body: "Module exports v1 with no validation.", recommendation: "Add validation." + }], + next_steps: [] + }) + }, + turnError: { message: "Reconnecting... 1/5" } + }); + + const result = runCompanion( + ["adversarial-review", "--base", "main", "--scope", "branch", "--cwd", cwd, "--json"], + fake.env + ); + + assert.equal(result.status, 0, "a recovered run with a valid verdict must exit success, not propagate the stale transient error status"); + const payload = JSON.parse(result.stdout.trim()); + assert.notEqual(payload.failed, true, "a recovered finalize turn must NOT be flagged failed"); + assert.equal(payload.result?.verdict, "needs-attention", "valid verdict must be preserved"); + assert.equal(payload.parseError, null, "valid JSON should parse cleanly"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +// ------------------------------------------------------------------- +// Unit tests for runAppServerInvestigation (continued from above) +// ------------------------------------------------------------------- + +test("outputSchema-set finalize turn produces schema-conformant final message", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: commands, no final answer + fake.queueTurnResponse({ + commands: [{ command: "git diff HEAD~1", exitCode: 0 }], + finalAnswer: null + }); + // Recon turn 2: no commands, final answer => converges + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "Investigation done." } + }); + // Finalize turn with structured output + fake.queueTurnResponse({ + finalAnswer: { text: STRUCTURED_REVIEW } + }); + + const schema = { + type: "object", + required: ["verdict"], + properties: { + verdict: { type: "string" }, + summary: { type: "string" }, + findings: { type: "array" }, + next_steps: { type: "array" } + } + }; + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate the changes.", + finalizePrompt: "Produce your structured verdict.", + outputSchema: schema + }); + + // The finalMessage should be parseable JSON with a verdict field + const parsed = JSON.parse(result.finalMessage); + assert.equal(parsed.verdict, "needs-attention"); + assert.equal(parsed.findings.length, 1); + assert.equal(parsed.findings[0].severity, "high"); + + // The finalize turn should have received the schema + const requests = fake.requests; + const starts = requests.filter((r) => r.method === "turn/start"); + assert.deepEqual(starts[starts.length - 1].params.outputSchema, schema); + + // Only the finalize turn's reasoningSummary is returned + assert.ok(Array.isArray(result.reasoningSummary)); + } finally { + fake.close(); + } +}); + +// ------------------------------------------------------------------- +// Task 5: renderer truncation banner +// ------------------------------------------------------------------- + +import { renderReviewResult } from "../plugins/codex/scripts/lib/render.mjs"; + +test("renderer prepends truncation banner when investigation.truncated is true", () => { + const parsed = { + parsed: { + verdict: "needs-attention", + summary: "Risk identified.", + findings: [], + next_steps: [] + } + }; + const out = renderReviewResult(parsed, { + reviewLabel: "Adversarial Review", + targetLabel: "branch:feature", + reasoningSummary: [], + investigation: { turnCount: 10, truncated: true } + }); + assert.match(out, /Investigation truncated at 10 turns; findings may be shallow\./); +}); + +test("renderer omits truncation banner when investigation is null or not truncated", () => { + const parsed = { + parsed: { + verdict: "approve", + summary: "Looks fine.", + findings: [], + next_steps: [] + } + }; + const outNull = renderReviewResult(parsed, { + reviewLabel: "Adversarial Review", + targetLabel: "branch:feature", + reasoningSummary: [], + investigation: null + }); + assert.doesNotMatch(outNull, /Investigation truncated/); + + const outOk = renderReviewResult(parsed, { + reviewLabel: "Adversarial Review", + targetLabel: "branch:feature", + reasoningSummary: [], + investigation: { turnCount: 4, truncated: false } + }); + assert.doesNotMatch(outOk, /Investigation truncated/); +}); + +test("renderer shows truncation banner on parse-error and validation-error paths", () => { + const parseErrorOut = renderReviewResult( + { parsed: null, parseError: "Unexpected token", rawOutput: "{not json" }, + { + reviewLabel: "Adversarial Review", + targetLabel: "branch:feature", + reasoningSummary: [], + investigation: { turnCount: 10, truncated: true } + } + ); + assert.match(parseErrorOut, /Investigation truncated at 10 turns/, + "banner must appear when output is unparseable AND investigation was truncated"); + + const validationErrorOut = renderReviewResult( + { parsed: { not: "review-shaped" } }, + { + reviewLabel: "Adversarial Review", + targetLabel: "branch:feature", + reasoningSummary: [], + investigation: { turnCount: 10, truncated: true } + } + ); + assert.match(validationErrorOut, /Investigation truncated at 10 turns/, + "banner must appear when output has wrong shape AND investigation was truncated"); +}); + +// ------------------------------------------------------------------- +// Task 6: --max-investigation-turns CLI flag +// ------------------------------------------------------------------- + +import { parseArgs } from "../plugins/codex/scripts/lib/args.mjs"; + +test("parseArgs accepts --max-investigation-turns as a value option", () => { + const { options } = parseArgs( + ["--base", "main", "--max-investigation-turns", "15", "auth"], + { + valueOptions: ["base", "scope", "model", "cwd", "max-investigation-turns"], + booleanOptions: ["json", "background", "wait"] + } + ); + assert.equal(options["max-investigation-turns"], "15"); +}); + +test("--max-investigation-turns propagates from CLI to runAppServerInvestigation", async () => { + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + for (let i = 0; i < 6; i += 1) { + fake.queueTurnResponse({ + commands: [{ command: "git diff", exitCode: 0 }], + finalAnswer: null + }); + } + fake.queueTurnResponse({ + finalAnswer: { text: JSON.stringify({ verdict: "approve", summary: "ok", findings: [], next_steps: [] }) } + }); + + const result = runCompanion( + ["adversarial-review", + "--base", "main", "--scope", "branch", "--cwd", cwd, + "--max-investigation-turns", "5", + "--json"], + fake.env + ); + + assert.equal(result.status, 0, `expected exit 0, stderr: ${result.stderr}`); + const payload = JSON.parse(result.stdout.trim()); + assert.equal(payload.investigation.turnCount, 5); + assert.equal(payload.investigation.truncated, true); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("invalid --max-investigation-turns raises a clear error", () => { + const r = spawnSync("node", + [COMPANION_PATH, "adversarial-review", "--max-investigation-turns", "abc"], + { encoding: "utf8", timeout: 30000 } + ); + assert.notEqual(r.status, 0, "should exit non-zero on invalid flag value"); + assert.match( + r.stderr, + /must be a positive integer/, + "error message should explain the validation failure" + ); +}); + +test("--max-investigation-turns rejects malformed numeric tokens", () => { + // parseInt-style salvage must NOT accept these — they are typos, not valid input. + const cases = [ + "1.5", // parseInt would yield 1 + "10abc", // parseInt would yield 10 + "1e2", // exponential notation: Number(...) yields 100, but the contract is "integer literal" + " 5", // leading whitespace from accidental quoting + "0", // zero is not positive + "-3", // negative integer + "" // empty string + ]; + for (const value of cases) { + const r = spawnSync("node", + [COMPANION_PATH, "adversarial-review", "--max-investigation-turns", value], + { encoding: "utf8", timeout: 30000 } + ); + assert.notEqual(r.status, 0, `value ${JSON.stringify(value)} should exit non-zero`); + assert.match( + r.stderr, + /must be a positive integer/, + `value ${JSON.stringify(value)} should trigger the validation error` + ); + } +}); + +test("invalid --turn-idle-timeout raises a clear error", () => { + const r = spawnSync("node", + [COMPANION_PATH, "adversarial-review", "--turn-idle-timeout", "abc"], + { encoding: "utf8", timeout: 30000 } + ); + assert.notEqual(r.status, 0, "should exit non-zero on invalid flag value"); + assert.match(r.stderr, /must be a positive integer/, "error message should explain the validation failure"); +}); + +// ------------------------------------------------------------------- +// Prompt contract: inline + finalize prompts must forbid tool-call stubs +// ------------------------------------------------------------------- + +import { readFileSync } from "node:fs"; +import { fileURLToPath as fileURLToPathPromptCheck } from "node:url"; + +const PROMPT_DIR = fileURLToPathPromptCheck( + new URL("../plugins/codex/prompts/", import.meta.url) +); + +test("inline adversarial-review prompt forbids tool-call stub output", () => { + const text = readFileSync(`${PROMPT_DIR}adversarial-review.md`, "utf8"); + // The model used to respond with payloads like {"cmd": "wc -l ..."} + // instead of the review JSON — the prompt must explicitly disallow it + // since this path is single-turn and has no recovery. + assert.match(text, /tool[- ]?call|tool[- ]?use/i, + "inline prompt must mention tool-call/tool-use"); + assert.match(text, /\{"cmd"|stub/i, + "inline prompt must show or name the tool-call stub anti-pattern"); + assert.match(text, /Do NOT run any shell commands/, + "inline prompt must explicitly forbid running shell commands"); +}); + +test("finalize prompt forbids tool-call stub output", () => { + const text = readFileSync(`${PROMPT_DIR}adversarial-review-finalize.md`, "utf8"); + assert.match(text, /Do NOT run any shell commands/, + "finalize prompt must explicitly forbid running shell commands"); + assert.match(text, /no tool-call payloads|no shell commands/i, + "finalize prompt must spell out the no-tool-call rule"); +}); + +test("task turn that recovered from a transient reconnect is NOT marked failed (e2e)", async () => { + // A task turn emits a valid agent message AND a stale transient `error` + // ("Reconnecting... 1/5"), then turn/completed. The companion must exit 0 and + // report success — not propagate the stale non-zero status from buildResultStatus. + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + finalAnswer: { text: "ALLOW: looks fine" }, + turnError: { message: "Reconnecting... 1/5" } + }); + + const result = runCompanion(["task", "--json", "--cwd", cwd, "do the thing"], fake.env); + + assert.equal(result.status, 0, "a recovered task must exit 0, not propagate the stale transient error status"); + const payload = JSON.parse(result.stdout.trim()); + assert.match(payload.rawOutput, /ALLOW: looks fine/, "the real agent answer must be returned"); + assert.equal(payload.status, 0, "payload.status must be normalized to the resolved success status"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("task turn that errored with no usable output still fails (e2e)", async () => { + // Guard the genuine-failure case: a transient/fatal error with NO agent message + // must still exit non-zero. Otherwise resolveRunExitStatus would whitewash real + // failures. + const cwd = makeSelfCollectGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + finalAnswer: null, + turnError: { message: "Connection lost; giving up." } + }); + + const result = runCompanion(["task", "--json", "--cwd", cwd, "do the thing"], fake.env); + + assert.notEqual(result.status, 0, "a turn that errored with no usable output must still fail"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("native review that recovered from a transient reconnect is NOT marked failed (e2e)", async () => { + // The native /codex:review branch returns exitStatus from result.status raw. + // A recovered native review (reviewText present + stale error) must exit 0. + // Note: review uses reviewText as the usable-output signal, not finalMessage. + const cwd = makeInlineGitFixture(); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ + reviewText: "Reviewed current changes.\nNo material issues found.", + turnError: { message: "Reconnecting... 1/5" } + }); + + const result = runCompanion( + ["review", "--base", "main", "--scope", "branch", "--cwd", cwd, "--json"], + fake.env + ); + + assert.equal(result.status, 0, "a recovered native review with review text must exit 0"); + const payload = JSON.parse(result.stdout.trim()); + assert.equal(payload.codex.status, 0, "payload.codex.status must be normalized to the resolved success status"); + } finally { + fake.close(); + rmSync(cwd, { recursive: true, force: true }); + } +}); + +test("idle watchdog interrupts with the buffered turn id when the turn/start RPC reply is delayed (Defect C)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: progresses (has commands), does not converge. + fake.queueTurnResponse({ commands: [{ command: "git diff", exitCode: 0 }], finalAnswer: null }); + // Recon turn 2: emits turn/started, then withholds the RPC result and never + // completes — so the watchdog fires while state.turnId is still null. + fake.queueTurnHangAfterStarted(); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + turnIdleTimeoutMs: 300 + }); + + assert.ok(result.error, "a stalled turn must abort with an error"); + assert.match(result.error.message, /idle|timeout|timed out/i); + + // The fixture must have received a turn/interrupt carrying the turn id it + // announced via turn/started — proving the watchdog did not skip the + // interrupt just because state.turnId was null (Defect C). turn_2 is the + // hung turn's id (turn_1 was recon turn 1). The interrupt is fire-and-forget + // and lands on the fixture exactly as withAppServer's client.close() is + // resolving, so the fixture's synchronous saveState can land a beat after + // the call returns; poll the state file briefly for it to appear. + const statePath = path.join(fake.binDir, "fake-codex-state.json"); + let state = JSON.parse(fs.readFileSync(statePath, "utf8")); + const deadline = Date.now() + 2000; + while (!state.lastInterrupt && Date.now() < deadline) { + await new Promise((resolve) => setTimeout(resolve, 20)); + state = JSON.parse(fs.readFileSync(statePath, "utf8")); + } + assert.ok(state.lastInterrupt, "watchdog must send turn/interrupt even before the turn/start RPC reply"); + assert.equal(state.lastInterrupt.turnId, "turn_2", "interrupt must carry the buffered turn id"); + } finally { + fake.close(); + } +}); + +test("foreign-thread chatter does not re-arm the current turn's idle watchdog (Defect B)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: emits turn/started, then a long stream of foreign-thread + // notifications spaced UNDER the idle window, then never completes OUR turn. + // Spans ~4s so the buggy (re-arm-on-foreign) path cannot time out until + // chatter stops (~4s + the 300ms idle window ≈ 4.3s); the fixed path times + // out promptly ~300ms after turn/started. The 2500ms ceiling sits between + // the two with wide margins on both sides (≈1.8s of buggy-side headroom and + // ≈2s of tolerance for subprocess-startup jitter on the fixed side), so the + // assertion discriminates without being flaky on a loaded machine. + fake.queueTurnResponse({ foreignChatterThenHang: { count: 80, everyMs: 50 } }); + + const start = Date.now(); + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + turnIdleTimeoutMs: 300 + }); + const elapsed = Date.now() - start; + + assert.ok(result.error, "stuck turn must time out despite foreign chatter"); + assert.match(result.error.message, /idle|timeout|timed out/i); + assert.ok(elapsed < 2500, `watchdog must fire at the idle window, not be held open by foreign chatter (took ${elapsed}ms)`); + } finally { + fake.close(); + } +}); + +test("plain recon turn does not infer completion from a readiness cue (Defect A gate)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + // Recon turn 1: emits a "ready for finalize" final_answer cue, then goes + // silent — NO real turn/completed, NO subagent work. A plain turn must wait + // for turn/completed (which never arrives) -> idle watchdog aborts. + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "I'm ready for finalize." }, + cueThenHang: true + }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + // Quiet window well below the idle timeout: on UNFIXED code, cue-based + // inference (now using this window) would fire at 60ms and the run would + // not error. On FIXED code, a plain turn never infers, so only the idle + // watchdog (400ms) ends it -> result.error is set. + inferredCompletionQuietMs: 60, + turnIdleTimeoutMs: 400 + }); + + assert.ok(result.error, "a plain turn with no real turn/completed must time out, not infer"); + assert.match(result.error.message, /idle|timeout|timed out/i); + // Finalize must NOT have been dispatched (the recon turn never completed). + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 1, "finalize must not be dispatched when recon never completed"); + } finally { + fake.close(); + } +}); + +test("a verdict streamed after a readiness cue is captured, not discarded (Defect A end-to-end)", async () => { + const cwd = makeTempDir("codex-inv-test-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.enableSerialization(); + // Recon turn 1: a "ready for finalize" cue immediately, then the REAL verdict + // (~300ms) and the REAL turn/completed (~600ms). delayCompletedMs (600) is + // ABOVE the unfixed 250ms inference window, so unfixed code dispatches + // finalize while recon is still busy -> finalize hangs -> watchdog error. + fake.queueTurnResponse({ + commands: [], + finalAnswer: { text: "I'm ready for finalize." }, + lateFinalAnswer: { text: "Investigation complete. Verdict ready.", afterMs: 300 }, + delayCompletedMs: 600 + }); + // Finalize turn 2: schema-enforced structured JSON. + fake.queueTurnResponse({ finalAnswer: { text: STRUCTURED_REVIEW } }); + + const result = await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + turnIdleTimeoutMs: 2000 + // No inferredCompletionQuietMs override: a plain recon turn must never + // infer regardless of the window. The fix is the sawSubagentWork gate. + }); + + assert.equal(result.error ?? null, null, "fixed code must NOT hang to the watchdog"); + assert.equal(result.finalMessage, STRUCTURED_REVIEW, "verdict from finalize is preserved"); + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 2, "recon completes, then finalize is dispatched onto an idle thread"); + } finally { + fake.close(); + } +}); + +test("finalize turn downgrades to medium effort while investigation keeps caller effort", async () => { + const cwd = makeTempDir("codex-inv-effort-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + fake.queueTurnResponse({ finalAnswer: { text: APPROVE_REVIEW } }); + + await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + effort: "xhigh" + }); + + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 2); + assert.equal(starts[0].params.effort, "xhigh", "investigation turn keeps caller effort"); + assert.equal(starts[1].params.effort, "medium", "finalize turn downgrades to medium"); + } finally { + fake.close(); + } +}); + +test("CODEX_COMPANION_FINALIZE_EFFORT=inherit keeps caller effort on finalize", async () => { + process.env.CODEX_COMPANION_FINALIZE_EFFORT = "inherit"; + const cwd = makeTempDir("codex-inv-effort-inherit-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + fake.queueTurnResponse({ finalAnswer: { text: APPROVE_REVIEW } }); + + await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + effort: "xhigh" + }); + + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.at(-1).params.effort, "xhigh"); + } finally { + delete process.env.CODEX_COMPANION_FINALIZE_EFFORT; + fake.close(); + } +}); + +test("an explicit CODEX_COMPANION_FINALIZE_EFFORT value overrides the medium default", async () => { + process.env.CODEX_COMPANION_FINALIZE_EFFORT = "LOW"; + const cwd = makeTempDir("codex-inv-effort-explicit-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + fake.queueTurnResponse({ finalAnswer: { text: APPROVE_REVIEW } }); + + await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + effort: "xhigh" + }); + + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.at(-1).params.effort, "low", "value is normalized and honored"); + } finally { + delete process.env.CODEX_COMPANION_FINALIZE_EFFORT; + fake.close(); + } +}); + +test("an unrecognized CODEX_COMPANION_FINALIZE_EFFORT falls back to medium", async () => { + process.env.CODEX_COMPANION_FINALIZE_EFFORT = "turbo"; + const cwd = makeTempDir("codex-inv-effort-invalid-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + fake.queueTurnResponse({ finalAnswer: { text: APPROVE_REVIEW } }); + + await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + effort: "xhigh" + }); + + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.at(-1).params.effort, "medium"); + } finally { + delete process.env.CODEX_COMPANION_FINALIZE_EFFORT; + fake.close(); + } +}); + +test("finalize retry also uses the downgraded effort", async () => { + const cwd = makeTempDir("codex-inv-effort-retry-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + // First finalize violates the contract by running a command => strict retry. + fake.queueTurnResponse({ + commands: [{ command: "wc -l README.md", exitCode: 0 }], + finalAnswer: { text: "{\"cmd\":\"wc -l README.md\"}" } + }); + fake.queueTurnResponse({ commands: [], finalAnswer: { text: APPROVE_REVIEW } }); + + await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + effort: "xhigh" + }); + + const starts = fake.requests.filter((r) => r.method === "turn/start"); + assert.equal(starts.length, 3); + assert.equal(starts[1].params.effort, "medium"); + assert.equal(starts[2].params.effort, "medium"); + } finally { + fake.close(); + } +}); + +test("finalize effort is resolved per call, not cached at import time", async () => { + // Regression guard: reading the env var at module load would make the first + // run in a process pin the value for every later run. + const cwd = makeTempDir("codex-inv-effort-percall-"); + const fake = setupFakeCodex({ cwd }); + try { + fake.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + fake.queueTurnResponse({ finalAnswer: { text: APPROVE_REVIEW } }); + await runAppServerInvestigation(fake.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + effort: "xhigh" + }); + assert.equal(fake.requests.filter((r) => r.method === "turn/start").at(-1).params.effort, "medium"); + } finally { + fake.close(); + } + + process.env.CODEX_COMPANION_FINALIZE_EFFORT = "high"; + const cwd2 = makeTempDir("codex-inv-effort-percall-2-"); + const fake2 = setupFakeCodex({ cwd: cwd2 }); + try { + fake2.queueTurnResponse({ commands: [], finalAnswer: { text: "Investigation done." } }); + fake2.queueTurnResponse({ finalAnswer: { text: APPROVE_REVIEW } }); + await runAppServerInvestigation(fake2.cwd, { + investigatePrompt: "Investigate.", + finalizePrompt: "Finalize.", + outputSchema: { type: "object", required: ["verdict"] }, + effort: "xhigh" + }); + assert.equal(fake2.requests.filter((r) => r.method === "turn/start").at(-1).params.effort, "high"); + } finally { + delete process.env.CODEX_COMPANION_FINALIZE_EFFORT; + fake2.close(); + } +}); diff --git a/tests/render.test.mjs b/tests/render.test.mjs index ab68038e5..40ee20a4a 100644 --- a/tests/render.test.mjs +++ b/tests/render.test.mjs @@ -27,6 +27,51 @@ test("renderReviewResult degrades gracefully when JSON is missing required revie assert.match(output, /Raw final message:/); }); +test("renderReviewResult reports run failure instead of faking a JSON parse error", () => { + // When the run failed at the transport/turn level (connection dropped, + // idle timeout), the leftover `rawOutput` is often just the model's opening + // line — NOT structured JSON. The renderer must report the real failure + // reason, not a misleading "did not return valid structured JSON" parse error. + const output = renderReviewResult( + { + parsed: null, + parseError: null, + failed: true, + failureMessage: "stream disconnected before completion: error sending request for url (https://bedrock-mantle.../responses)", + rawOutput: "Using `superpowers:writing-plans` to judge the plan before read-only investigation." + }, + { + reviewLabel: "Adversarial Review", + targetLabel: "branch diff" + } + ); + + assert.match(output, /could not complete/i, "should state the run could not complete"); + assert.match(output, /stream disconnected before completion/, "should surface the real failure reason"); + assert.doesNotMatch(output, /valid structured JSON/, "must NOT misrender as a JSON parse error"); + assert.doesNotMatch(output, /Parse error:/, "must NOT show a parse error for a transport failure"); +}); + +test("renderReviewResult still shows the parse-error path for genuine malformed JSON", () => { + // Regression guard: a status-0 run that legitimately returns malformed JSON + // (model formatting bug, not a transport failure) must keep the existing + // parse-error rendering. `failed` is absent here. + const output = renderReviewResult( + { + parsed: null, + parseError: "Unexpected token '{' ...", + rawOutput: "{not valid json" + }, + { + reviewLabel: "Adversarial Review", + targetLabel: "working tree diff" + } + ); + + assert.match(output, /did not return valid structured JSON/); + assert.match(output, /Parse error:/); +}); + test("renderStoredJobResult prefers rendered output for structured review jobs", () => { const output = renderStoredJobResult( { diff --git a/tests/runtime.test.mjs b/tests/runtime.test.mjs index 8f276835b..a090a047e 100644 --- a/tests/runtime.test.mjs +++ b/tests/runtime.test.mjs @@ -395,7 +395,13 @@ test("adversarial review accepts the same base-branch targeting as review", () = fs.writeFileSync(path.join(repo, "src", "app.js"), "export const value = items[0];\n"); run("git", ["add", "src/app.js"], { cwd: repo }); run("git", ["commit", "-m", "init"], { cwd: repo }); + // Commit the change on a feature branch so `--base main` resolves a NON-empty + // range. (A change left uncommitted on main is invisible to branch scope: + // merge-base == HEAD, the diff is empty, and there is nothing to review.) + run("git", ["checkout", "-b", "feature/base-target"], { cwd: repo }); fs.writeFileSync(path.join(repo, "src", "app.js"), "export const value = items[0].id;\n"); + run("git", ["add", "src/app.js"], { cwd: repo }); + run("git", ["commit", "-m", "tweak"], { cwd: repo }); const result = run("node", [SCRIPT, "adversarial-review", "--base", "main"], { cwd: repo, @@ -429,8 +435,14 @@ test("adversarial review asks Codex to inspect larger diffs itself", () => { assert.equal(result.status, 0, result.stderr); const state = JSON.parse(fs.readFileSync(path.join(binDir, "fake-codex-state.json"), "utf8")); - assert.match(state.lastTurnStart.prompt, /lightweight summary/i); - assert.match(state.lastTurnStart.prompt, /read-only git commands/i); + // With the two-phase investigation flow, the default fake converges on turn 1 + // (no commands, finalAnswer), then the finalize turn fires. lastTurnStart + // captures the finalize turn, which uses buildAdversarialFinalizePrompt. + // The investigate prompt (turn 1) contained the self-collect guidance; the + // finalize prompt (turn 2) references the investigation completed in prior turns. + assert.match(state.lastTurnStart.prompt, /investigation|structured review/i); + // The finalize prompt carries no repository context of its own — the diff was + // already established during the investigation turns it refers back to. assert.doesNotMatch(state.lastTurnStart.prompt, /PROMPT_SELF_COLLECT_[ABC]/); }); @@ -883,13 +895,36 @@ test("task can finish after subagent work even if the parent turn/completed even const result = run("node", [SCRIPT, "task", "challenge the current design"], { cwd: repo, - env: buildEnv(binDir) + // The subagent-completion fallback now uses a 15s default quiet window; + // shrink it via the env override so this test resolves in ms, not 15s. + env: { ...buildEnv(binDir), CODEX_INFERRED_COMPLETION_QUIET_MS: "50" } }); assert.equal(result.status, 0, result.stderr); assert.equal(result.stdout, "Handled the requested task.\nTask prompt accepted.\n"); }); +test("CODEX_INFERRED_COMPLETION_QUIET_MS overrides the inferred-completion quiet window", () => { + const repo = makeTempDir(); + const binDir = makeTempDir(); + installFakeCodex(binDir, "with-subagent-no-main-turn-completed"); + initGitRepo(repo); + fs.writeFileSync(path.join(repo, "README.md"), "hello\n"); + run("git", ["add", "README.md"], { cwd: repo }); + run("git", ["commit", "-m", "init"], { cwd: repo }); + + const start = Date.now(); + const result = run("node", [SCRIPT, "task", "challenge the current design"], { + cwd: repo, + env: { ...buildEnv(binDir), CODEX_INFERRED_COMPLETION_QUIET_MS: "50" } + }); + const elapsed = Date.now() - start; + + assert.equal(result.status, 0, result.stderr); + assert.equal(result.stdout, "Handled the requested task.\nTask prompt accepted.\n"); + assert.ok(elapsed < 10000, `inference must fire on the short window (took ${elapsed}ms)`); +}); + test("task using the shared broker still completes when Codex spawns subagents", () => { const repo = makeTempDir(); const binDir = makeTempDir(); @@ -2061,6 +2096,38 @@ test("stop hook allows the stop when the review gate is enabled and the stop-tim assert.equal(allowed.stdout.trim(), ""); }); +test("stop hook parses the ALLOW answer when the stop-time review task recovered from a transient error", () => { + // Regression: a gate review that survives a transient "Reconnecting..." notice + // still completes with a valid ALLOW answer. The task must exit 0 so the hook + // parses ALLOW/BLOCK instead of false-positive blocking on "task failed". + const repo = makeTempDir(); + const binDir = makeTempDir(); + installFakeCodex(binDir, "gate-recovered"); + initGitRepo(repo); + fs.writeFileSync(path.join(repo, "README.md"), "hello\n"); + run("git", ["add", "README.md"], { cwd: repo }); + run("git", ["commit", "-m", "init"], { cwd: repo }); + + const setup = run("node", [SCRIPT, "setup", "--enable-review-gate", "--json"], { + cwd: repo, + env: buildEnv(binDir) + }); + assert.equal(setup.status, 0, setup.stderr); + + const result = run("node", [STOP_HOOK], { + cwd: repo, + env: buildEnv(binDir), + input: JSON.stringify({ + cwd: repo, + session_id: "sess-stop-recovered", + last_assistant_message: "I completed the refactor." + }) + }); + + assert.equal(result.status, 0, result.stderr); + assert.equal(result.stdout.trim(), "", "a recovered ALLOW review must NOT block the session"); +}); + test("stop hook does not block when Codex is unavailable even if the review gate is enabled", () => { const repo = makeTempDir(); initGitRepo(repo);