v0.20.0: Retained Spatial UI, Unified Interaction Pipeline, 3D Object Detection & Off-Thread Vision

XR Blocks v0.20.0 Release Notes

We are excited to announce XR Blocks 0.20.0, a breaking release that simplifies and aligns the interaction and UI authoring interfaces. This release simplifies the input, hit testing, manipulation, reticles, and UI manipulation into one shared runtime model. The simulator, examples, templates, documentation, build output, and agent workflows have all been updated to be more agent-native. The release also introduces off-thread vision inference, reusable 3D object detection, lazy build chunks, and experimental spatial anchor support.

---

Highlights

• Retained Spatial UI: Built-in flex-layout UI system powered by @pmndrs/uikit using native THREE.Object3D hierarchies (UICard, UIOverlay, UIPanel, UIButton, UISlider, UIText, UIImage, UIIcon).
• Unified Interaction & Manipulation Pipeline: Single hit-testing and event pipeline across mouse, gaze, controllers, hand rays, and direct touch, paired with an object.xb.manipulation contract for translate, rotate, and scale.
• Off-Thread Vision Processing: MediaPipe pose and face inference run in background Web Workers, with 3D metric skeleton placement and joint confidence filtering.
• 3D Object Detection Add-on: New objects3d add-on combining 2D segmentation (MediaPipe/SlimSAM) with depth sampling, oriented bounding box (OBB) fitting, and multi-view temporal fusion.
• Lazy Module Splitting: UI and simulator browser modules split into lazy chunks (build/internal/), keeping core bundles lightweight.
• Asynchronous & Terminal Lifecycle: Safe async script initialization, robust error propagation, and terminal Core.dispose() with full resource cleanup.

---

Major Subsystem Updates

Retained Spatial UI

• Powered by @pmndrs/uikit under the hood and composed directly via Object3D parenting; no extra UI managers or separate raycasters required.
• UICard serves as the world-space root (meter-based sizing, anchors, manipulation) and UIOverlay serves as the view-space root (excluded from depth maps).
• Rich flexbox styling (UIStyle), pseudo-states (:hover, :active, :disabled), adaptive text, and six built-in theme presets (grayGlass, glimmer, colorful, etc.).
• Built-in layout and bounds validation via xb.ui.validate().

Unified Interaction, Manipulation & Placement

• Interaction events (source, target, surface, intersection) bubble through the target hierarchy; propagation is controlled via event.stopPropagation().
• Standardized object configuration via object.xb (pointerEvents, interactionEnabled, reticleMode, manipulation).
• Public placement scripts added for common behaviors: FaceCamera, FollowHead, FollowObject, and Orbit.

World Sensing & Performance

• MediaPipe Web Workers: Pose and face tracking run off-thread via MediaPipeVisionWorker, preventing main-thread drops.
• Pose Landmarks: Positions body skeletons using real-world 3D metric landmarks with customizable confidence thresholds (minJointVisibility, minJointPresence).
• Depth & Occlusion: matchDepthView disabled by default in OcclusionOptions for direct camera pose alignment; duplicate second-eye and unchanged simulator depth readbacks are skipped.
• Simulation Clock & Physics: getElapsedTime('simulation' | 'render') freezes simulation time during pause; manual frame stepping accumulates fixed physics steps accurately.

Add-ons & ModelViewer

• objects3d: Detects 3D oriented bounding boxes using 2D masks (MediaPipe/SlimSAM), depth projection, and multi-view temporal clustering.
• virtualkeyboard: Rebuilt with retained UI elements, support for full 3D spatial transforms, and a new standalone sample.
• ModelViewer: Rebuilt on the unified interaction and UI pipeline, supporting .gltf, .glb, .ply, .spz, .splat, and .ksplat with normalized manipulation proxies and animation control.

Spatial Anchors (Quest 3 & Simulator)

• Added xb.world.anchors (AnchorManager) and AnchoredObjects for anchoring and persisting scene objects across sessions.
• Includes capability probing (AnchorCapability), storage management (LocalStorageAnchorStore), and desktop fallback (SimulatorAnchor).
• Note: Persistence is currently supported on Meta Quest 3 (and in the desktop simulator); Android XR persistence will become available once supported by the Chrome WebXR implementation.

---

Breaking Changes & Migration

Removed / Changed	Replacement
uiblocks entry & legacy UI (View, Panel, SpatialPanel, TextView, etc.)	Import UICard, UIOverlay, UIButton, UIText directly from xrblocks.
Options.enableUI(), Options.uikit, UIKitOptions	UI is initialized automatically on demand when adding UI elements to the scene.
DragManager, DragMode, Script.ux, draggable flags	Configure object.xb.manipulation = { translate: true, rotate: true, scale: true }.
ignoreReticleRaycast, custom raycasters	Use object.xb.pointerEvents, object.xb.reticleMode, and xb.user raycast queries.
SelectEvent.target was the controller	event.target is now the logical target object. Physical input is on event.source.controller.
Callback returning true to stop propagation	Call event.stopPropagation().
Synchronous Core.dispose()	await xb.core.dispose() is asynchronous and terminal.
Single-file bundle distribution	Deploy the full build/ directory including lazy build/internal/ chunks.

👉 For step-by-step instructions, see the complete Migrating to v0.20.0 Guide.

What's Changed

• Bump actions/setup-node from 6.5.0 to 7.0.0 in the ci-actions group across 1 directory by @dependabot[bot] in #472
• Add XREAL Aura by @salmanmkc in #488
• feat(addons/objects3d): reusable 3D object detection (2D detect + depth into oriented boxes) by @salmanmkc in #417
• perf(world/humans): run MediaPipe PoseLandmarker in a web worker by @salmanmkc in #489
• Disable matchDepthView by default and update occlusion by @dli7319 in #491
• fix(virtualkeyboard): honour the keyboard transform by @salmanmkc in #495
• place the pose skeleton from MediaPipe's metric landmarks by @salmanmkc in #503
• skip the second eye's depth readback by @salmanmkc in #507
• clear the physics interval and free the rapier world on dispose by @salmanmkc in #506
• skip the simulator depth readback when nothing has changed by @salmanmkc in #505
• fix(evals): run the prototype sweeps on Windows by @salmanmkc in #510
• feat(world/humans): let callers reject pose joints the model only guessed by @salmanmkc in #504
• fix(evals): stop comments satisfying an expected API by @salmanmkc in #509
• feat(mcp): add a server exposing the xb-* skills and API surface by @salmanmkc in #508
• feat(evals): cover head gestures, lipsync, context and agent hands by @salmanmkc in #514
• build(deps): bump js-yaml from 4.3.0 to 4.3.1 in /docs in the npm_and_yarn group across 1 directory by @dependabot[bot] in #511
• event.stopPropagation() on callbacks by @whuang37 in #512
• Realigned UI Rendering Backend by @whuang37 in #513
• v0.20 by @whuang37 in #517
• fixed packagelock dependencies by @whuang37 in #518
• Update brace-expansion and nanoid by @dli7319 in #519
• Migrated Virtual Keyboard by @whuang37 in #520
• feat(world/anchors): persistent spatial anchors by @salmanmkc in #494
• Realigned Physics Stepping When Pause by @whuang37 in #521
• created migration guide by @whuang37 in #522

Full Changelog: v0.19.0...v0.20.0