feat(evals): add AI-enabled V4 code surface#2378
Conversation
|
8440835 to
f919f92
Compare
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found and verified against the latest diff
Confidence score: 3/5
- In
packages/evals/core/contracts/tool.ts, addingtoolSurface: "v4_code"toToolSurfacelets core callers pass a value the core registry cannot initialize, which can cause runtime init failures for otherwise valid-looking configs; remove this harness-only surface from the core contract (or add a matching core registry implementation) before merging.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/core/contracts/tool.ts">
<violation number="1" location="packages/evals/core/contracts/tool.ts:23">
P2: Core callers can now pass `toolSurface: "v4_code"` as a valid contract value, but initialization immediately fails because the core registry has no implementation. Keep this harness-only surface out of `ToolSurface`, or register/support it in the core path.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| | "understudy_code" | ||
| | "playwright_code" | ||
| | "cdp_code" | ||
| | "v4_code" |
There was a problem hiding this comment.
P2: Core callers can now pass toolSurface: "v4_code" as a valid contract value, but initialization immediately fails because the core registry has no implementation. Keep this harness-only surface out of ToolSurface, or register/support it in the core path.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/contracts/tool.ts, line 23:
<comment>Core callers can now pass `toolSurface: "v4_code"` as a valid contract value, but initialization immediately fails because the core registry has no implementation. Keep this harness-only surface out of `ToolSurface`, or register/support it in the core path.</comment>
<file context>
@@ -20,6 +20,7 @@ export type ToolSurface =
| "understudy_code"
| "playwright_code"
| "cdp_code"
+ | "v4_code"
| "v4_code_deterministic"
| "playwright_mcp"
</file context>
There was a problem hiding this comment.
The shared ToolSurface type is intentionally the selection contract across core and external harnesses. listCoreTools()/getCoreTool() define the core-only subset, while resolveClaudeCodeToolSurface() owns this external-harness dispatch. Keeping v4_code in the shared union is required for typed harness rows and does not register it as a core tool.
There was a problem hiding this comment.
The parent comment was too broad here: ToolSurface is intentionally shared, and v4_code is not registered as a core tool because listCoreTools()/getCoreTool() and resolveClaudeCodeToolSurface() split those responsibilities. The issue is the core registry boundary, not the shared union itself.
fafeac6 to
f70ac5c
Compare
f70ac5c to
3643ee2
Compare
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Confidence score: 3/5
- In
packages/evals/framework/claudeCodeToolAdapter.ts, the MCP tool description is now out of sync with the prompt (it still referencespage.act/extract/observeand missesstagehand), which can steer Claude toward invalid tool calls and reduce eval reliability; align the tool description/schema with the injected MCP format to de-risk run behavior.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/framework/claudeCodeToolAdapter.ts">
<violation number="1" location="packages/evals/framework/claudeCodeToolAdapter.ts:1383">
P2: AI runs receive contradictory tool guidance: the MCP tool still says `page.act/extract/observe` exist and omits `stagehand` from scope. Update that tool description/schema alongside this prompt so Claude uses the injected `stagehand` facade rather than failing calls on `page`.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| "The context facade supports pages, newPage, activePage, setActivePage, clipboard, addInitScript, setExtraHTTPHeaders, domain policy, and cookie methods. context.close is intentionally unavailable because the harness owns cleanup.", | ||
| "AI methods act, extract, and observe are intentionally omitted from the provided facade. Stagehand internals and RPC objects are also omitted. This is an API-surface distinction, not a hostile-code security sandbox.", | ||
| mode === "ai" | ||
| ? "AI methods belong to the stagehand facade: await stagehand.act(instruction, options?), await stagehand.observe(instruction?, options?), and await stagehand.extract(instruction, schema, options?). Build extraction schemas with z, for example: await stagehand.extract('Extract the heading', z.object({ heading: z.string() })). To target a non-active page, pass its facade in options, for example { page }." |
There was a problem hiding this comment.
P2: AI runs receive contradictory tool guidance: the MCP tool still says page.act/extract/observe exist and omits stagehand from scope. Update that tool description/schema alongside this prompt so Claude uses the injected stagehand facade rather than failing calls on page.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/claudeCodeToolAdapter.ts, line 1383:
<comment>AI runs receive contradictory tool guidance: the MCP tool still says `page.act/extract/observe` exist and omits `stagehand` from scope. Update that tool description/schema alongside this prompt so Claude uses the injected `stagehand` facade rather than failing calls on `page`.</comment>
<file context>
@@ -1374,14 +1374,14 @@ function buildV4CodePromptInstructions(
mode === "ai"
- ? "AI methods use the exact positional signatures await page.act(instruction), await page.observe(instruction), and await page.extract(instruction, schema). Build extraction schemas with z, for example: await page.extract('Extract the heading', z.object({ heading: z.string() }))."
- : "AI methods act, extract, and observe are intentionally omitted from the provided facade. No model or AI provider credential is configured in this mode.",
+ ? "AI methods belong to the stagehand facade: await stagehand.act(instruction, options?), await stagehand.observe(instruction?, options?), and await stagehand.extract(instruction, schema, options?). Build extraction schemas with z, for example: await stagehand.extract('Extract the heading', z.object({ heading: z.string() })). To target a non-active page, pass its facade in options, for example { page }."
+ : "The stagehand facade, z, and AI methods act, extract, and observe are intentionally omitted. No model or AI provider credential is configured in this mode.",
"Stagehand internals and RPC objects are omitted. This is an API-surface distinction, not a hostile-code security sandbox.",
</file context>
2 participants
Summary
v4_codeas the AI-enabled companion tov4_code_deterministicpage.act(instruction),page.observe(instruction), andpage.extract(instruction, schema)methods only in AI mode, withzavailable for extraction schemasThis is the second draft in a stack and is based on the deterministic local V4 code surface. Both modes use the same extension-backed local browser runtime; the added AI methods are the intentional experimental variable.
Linear: https://linear.app/browserbase/issue/STG-2662/add-ai-enabled-v4-code-mode-to-stagehand-evals
E2E Test Matrix
extract,observe, andactextractreturned the page heading,observereturned an available action, andactsuccessfully navigated to the requested destination; the recorded Chrome process was gone after cleanup.VERIFIER_DISABLE_TRUNCATION=1 node packages/evals/dist/cli/cli.js run b:webtailbench --harness claude_code --tool v4_code --startup tool_launch_local --env local --model anthropic/claude-sonnet-5 --trials 1 --concurrency 1 --filter ids=recreation.gov_question_answering_26 --success outcomewithSTAGEHAND_V4_SDK_PATH=<local V4 SDK entry>page.act, completed the live browser task, and the outcome verifier accepted the result; cleanup left no V4 bridge or Chrome process.--tool v4_code_deterministicpnpm exec turbo run build:esm build:cli --filter=@browserbasehq/stagehand-evals --forcepnpm --filter @browserbasehq/stagehand-evals exec vitest runpnpm --filter @browserbasehq/stagehand-evals lintRelease
No changeset: this is private eval scaffolding and does not change the published Stagehand package runtime.
Summary by cubic
Adds
v4_code, a Stagehand V4 surface that runs with a selected model, alongsidev4_code_deterministic. The Claude Code harness now passes typed init IPC withmodeandmodel, validates model identifiers and credentials, injectsz(Zod), and exposesstagehand.act/observe/extractin AI mode while keeping the page/context facades deterministic and unchanged. Addresses Linear STG-2662.New Features
v4_code(model-enabled) andv4_code_deterministicin LOCALtool_launch_local; bench harness forwards--model.--model: normalize names, infer provider when omitted, reject unsupported providers, require provider API keys, and add Anthropicanthropic-dangerous-direct-browser-access.modeandmodel; Stagehand constructor receivesmodelonly in model-enabled runs; typed init IPC enforces this.stagehandfacade (stagehand.act/observe/extract(schema)), withzavailable for schemas; deterministic mode rejects model config.stagehand-scoped methods.Migration
--tool v4_code --startup tool_launch_local --env local --model <provider/model>, e.g.anthropic/claude-sonnet-5.ANTHROPIC_API_KEY,OPENAI_API_KEY,GEMINI_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY/GOOGLE_API_KEY,GROQ_API_KEY,CEREBRAS_API_KEY.STAGEHAND_V4_SDK_PATHpoints to the local V4 SDK.Written for commit c53abd7. Summary will update on new commits.