Contrib/2384#2385
Draft
miguelg719 wants to merge 8 commits into
Draft
Conversation
The computer-use agent actuates every click via `Input.dispatchMouseEvent` (page.click). On a mobile session the browser renders the site's touch-gated mobile layout, whose handlers only respond to touch/pointer events — so a synthesized mouse click does not register (e.g. an SFCC size selector keeps showing "please choose a size" and add-to-cart fails). Add a coordinate `page.tap(x, y)` (trusted `Input.dispatchTouchEvent`, mirroring `page.click`) and route the CUA `click` action to it when the session presents as mobile (`navigator.userAgentData.mobile`, detected once per run). Desktop sessions are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…or replay - Route only a single LEFT click to tap; right/middle/multi-click keep the mouse path (preserves context-menu and double-click behavior). - On a recording (replay) mobile run, record the tap as a deterministic "tap" step (page.tap gains returnXpath) and add a replayable tap method: locator.tap() + METHOD_HANDLER_MAP.tap. Not added to SupportedUnderstudyAction — replay dispatches by map lookup, so the LLM is not offered tap during live inference. - page.tap now updates the cursor overlay like other coordinate actions. - Add an integration test for tap actuation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: c77ad1e The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
miguelg719
marked this pull request as draft
July 23, 2026 08:05
miguelg719
commented
Jul 23, 2026
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
3 issues found and verified against the latest diff
Confidence score: 3/5
- In
packages/core/lib/v3/handlers/v3CuaAgentHandler.ts, the agent cache can replay a mobiletapin a later desktop session, which can send the wrong pointer type through gated flows and cause behavior divergence without a cache miss — include session/device mode in the cache key (or otherwise partition cache entries by pointer context). - Coverage is thin around tap/click dispatch across
packages/core/lib/v3/understudy/page.tsandpackages/core/lib/v3/handlers/v3CuaAgentHandler.ts, so a brokenPage.tappath or incorrect mobile/non-mobile routing/recording could ship silently — add focused integration and handler-level regression tests asserting both dispatch (page.tapvspage.click) and recorded action type for each mode.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/understudy/page.ts">
<violation number="1" location="packages/core/lib/v3/understudy/page.ts:1909">
P3: Coordinate taps have no focused regression coverage, so a broken `Page.tap` dispatch path or a mouse-typed event would ship unnoticed. Add an integration test that calls `page.tap(x, y)` and asserts the target receives touch/pointer-touch events.
(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>
<file name="packages/core/lib/v3/handlers/v3CuaAgentHandler.ts">
<violation number="1" location="packages/core/lib/v3/handlers/v3CuaAgentHandler.ts:358">
P3: This new CUA dispatch branch has no handler-level regression coverage for mobile/non-mobile selection or recording. Add focused mocks asserting `page.tap`/`page.click` and recorded `tap`/`click` behavior for both paths.
(Based on your team's feedback about unit tests for new behavior.) [FEEDBACK_USED]</violation>
<violation number="2" location="packages/core/lib/v3/handlers/v3CuaAgentHandler.ts:388">
P2: A local cache entry recorded on mobile replays as a touch action in a later desktop session, so pointer-type-gated flows can diverge without a cache miss. Include session/device mode in the agent-cache key, or choose tap versus click at replay time.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| const stagehandAction: Action = { | ||
| selector: normalized, | ||
| description: this.describePointerAction("tap", x, y), | ||
| method: "tap", |
Contributor
There was a problem hiding this comment.
P2: A local cache entry recorded on mobile replays as a touch action in a later desktop session, so pointer-type-gated flows can diverge without a cache miss. Include session/device mode in the agent-cache key, or choose tap versus click at replay time.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/handlers/v3CuaAgentHandler.ts, line 388:
<comment>A local cache entry recorded on mobile replays as a touch action in a later desktop session, so pointer-type-gated flows can diverge without a cache miss. Include session/device mode in the agent-cache key, or choose tap versus click at replay time.</comment>
<file context>
@@ -344,6 +347,59 @@ export class V3CuaAgentHandler {
+ const stagehandAction: Action = {
+ selector: normalized,
+ description: this.describePointerAction("tap", x, y),
+ method: "tap",
+ arguments: [],
+ };
</file context>
| * origin (top-left). Does not scroll. Requires a touch-capable (e.g. mobile) session. | ||
| */ | ||
| @FlowLogger.wrapWithLogging({ eventType: "PageTap" }) | ||
| async tap( |
Contributor
There was a problem hiding this comment.
P3: Coordinate taps have no focused regression coverage, so a broken Page.tap dispatch path or a mouse-typed event would ship unnoticed. Add an integration test that calls page.tap(x, y) and asserts the target receives touch/pointer-touch events.
(Based on your team's feedback about adding unit tests for new behavior.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/understudy/page.ts, line 1909:
<comment>Coordinate taps have no focused regression coverage, so a broken `Page.tap` dispatch path or a mouse-typed event would ship unnoticed. Add an integration test that calls `page.tap(x, y)` and asserts the target receives touch/pointer-touch events.
(Based on your team's feedback about adding unit tests for new behavior.) </comment>
<file context>
@@ -1897,6 +1897,49 @@ export class Page {
+ * origin (top-left). Does not scroll. Requires a touch-capable (e.g. mobile) session.
+ */
+ @FlowLogger.wrapWithLogging({ eventType: "PageTap" })
+ async tap(
+ x: number,
+ y: number,
</file context>
| // (context menu) and multi-click stay on the mouse path unchanged. Detected | ||
| // once via userAgentData (Browserbase os:"mobile" reports maxTouchPoints:0 | ||
| // but userAgentData.mobile:true). | ||
| const isPrimarySingleClick = |
Contributor
There was a problem hiding this comment.
P3: This new CUA dispatch branch has no handler-level regression coverage for mobile/non-mobile selection or recording. Add focused mocks asserting page.tap/page.click and recorded tap/click behavior for both paths.
(Based on your team's feedback about unit tests for new behavior.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/handlers/v3CuaAgentHandler.ts, line 358:
<comment>This new CUA dispatch branch has no handler-level regression coverage for mobile/non-mobile selection or recording. Add focused mocks asserting `page.tap`/`page.click` and recorded `tap`/`click` behavior for both paths.
(Based on your team's feedback about unit tests for new behavior.) </comment>
<file context>
@@ -344,6 +347,59 @@ export class V3CuaAgentHandler {
+ // (context menu) and multi-click stay on the mouse path unchanged. Detected
+ // once via userAgentData (Browserbase os:"mobile" reports maxTouchPoints:0
+ // but userAgentData.mobile:true).
+ const isPrimarySingleClick =
+ button === "left" && ((clickCount as number) ?? 1) === 1;
+ if (isPrimarySingleClick) {
</file context>
Replace the lazy per-run probe in the CUA click handler (a page.evaluate of
navigator.userAgentData.mobile, cached on first click) with a `usesTouch` getter
resolved from configuration:
- explicit `useTouch` wins, so callers can force either mode;
- otherwise derived from Browserbase `browserSettings.os` ("mobile"/"tablet") or
a local session's `localBrowserLaunchOptions.hasTouch`.
The probe had three problems the config signal does not: it could not run before
the first click, `userAgentData` is absent on insecure origins and on about:blank
(so a first click there cached a false negative for the whole run), and
`userAgentData.mobile` is false for tablets. It also cost a round trip per run
and made behavior depend on which page happened to be loaded first.
Note `os` is unavailable when resuming via browserbaseSessionID; `useTouch` is
the explicit opt-in for that case.
Adds unit coverage for the resolution matrix, which needs no browser.
The CUA handler was the only path actuating touch, but hybrid mode goes through V3AgentHandler and its own tools, so it still sent mouse clicks. Hybrid is the auto-selected mode for Claude / GPT-5.4 / Gemini 3, i.e. the default most callers get -- and "CUA hybrid agent (mouse): blocked" was a row in the original bug report, so the reported failure survived the first fix. Route on v3.usesTouch in: - clickTool, and record the step as "tap" so a cached mobile run does not replay as a mouse click; - typeTool and fillFormVisionTool, whose click-to-focus has the same problem -- an unregistered focus click means the text goes nowhere. Their recorded step stays a "type", which replays via the selector rather than a pointer event. Deliberately unchanged: double/triple click, right/middle click, drag and hover. Real touch drag needs touchMove interpolation rather than a routing switch. Adds unit coverage asserting each tool picks tap vs click and records the matching method.
Format with prettier and drop the three `any`s, which were failing the Lint job and -- because lint failure cancels the rest of the workflow -- taking every other check on the branch down with it. Inherited from #2373 on main; the eslint errors were latent there because `format:check` runs first and exited before eslint ever saw the file. Types the CDP payload the reducer actually reads instead of indexing an `any` record. Behavior is unchanged: the added `?? ""` only feeds `includes()`, where `undefined` and `""` are both misses, and the new optional chains sit behind guards that already established the object exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
why
what changed
test plan
Summary by cubic
Route both CUA and hybrid coordinate clicks to trusted touch taps on touch-capable sessions to fix touch-gated mobile UIs, with a config-driven
useTouchtoggle and deterministic tap record/replay.New Features
useTouchoption andv3.usesTouchresolution; explicit override or derived from BrowserbasebrowserSettings.os("mobile"/"tablet") or localhasTouch.page.tap(x, y)andlocator.tap()using trustedInput.dispatchTouchEvent; supports{ returnXpath }, updates the cursor overlay, andtapis registered in the act handler map for replay.clicktotapon touch sessions; Hybrid toolsclick,type, andfillFormVisionnow usetapfor coordinate focus/activation. Right/middle/multi-click, hover, drag remain mouse-based.tap; typing tools still recordtype. Added unit tests for touch resolution and tool routing, plus an integration test validating tap activation.Bug Fixes
reduce-logsby adding typed CDP payload (CdpParams) and formatting; no behavior change.Written for commit c77ad1e. Summary will update on new commits.