v0.20.0: Retained Spatial UI, Unified Interaction Pipeline, 3D Object Detection & Off-Thread Vision

XR Blocks v0.20.0 Release Notes

We are excited to announce XR Blocks 0.20.0, a breaking release that simplifies and aligns the interaction and UI authoring interfaces. This release simplifies the input, hit testing, manipulation, reticles, and UI manipulation into one shared runtime model. The simulator, examples, templates, documentation, build output, and agent workflows have all been updated to be more agent-native. The release also introduces off-thread vision inference, reusable 3D object detection, lazy build chunks, and experimental spatial anchor support.

---

Highlights

• Retained Spatial UI: Built-in flex-layout UI system powered by @pmndrs/uikit using native THREE.Object3D hierarchies (UICard, UIOverlay, UIPanel, UIButton, UISlider, UIText, UIImage, UIIcon).
• Unified Interaction & Manipulation Pipeline: Single hit-testing and event pipeline across mouse, gaze, controllers, hand rays, and direct touch, paired with an object.xb.manipulation contract for translate, rotate, and scale.
• Off-Thread Vision Processing: MediaPipe pose and face inference run in background Web Workers, with 3D metric skeleton placement and joint confidence filtering.
• 3D Object Detection Add-on: New objects3d add-on combining 2D segmentation (MediaPipe/SlimSAM) with depth sampling, oriented bounding box (OBB) fitting, and multi-view temporal fusion.
• Lazy Module Splitting: UI and simulator browser modules split into lazy chunks (build/internal/), keeping core bundles lightweight.
• Asynchronous & Terminal Lifecycle: Safe async script initialization, robust error propagation, and terminal Core.dispose() with full resource cleanup.

---

Major Subsystem Updates

Retained Spatial UI

• Powered by @pmndrs/uikit under the hood and composed directly via Object3D parenting; no extra UI managers or separate raycasters required.
• UICard serves as the world-space root (meter-based sizing, anchors, manipulation) and UIOverlay serves as the view-space root (excluded from depth maps).
• Rich flexbox styling (UIStyle), pseudo-states (:hover, :active, :disabled), adaptive text, and six built-in theme presets (grayGlass, glimmer, colorful, etc.).
• Built-in layout and bounds validation via xb.ui.validate().

Unified Interaction, Manipulation & Placement

• Interaction events (source, target, surface, intersection) bubble through the target hierarchy; propagation is controlled via event.stopPropagation().
• Standardized object configuration via object.xb (pointerEvents, interactionEnabled, reticleMode, manipulation).
• Public placement scripts added for common behaviors: FaceCamera, FollowHead, FollowObject, and Orbit.

World Sensing & Performance

• MediaPipe Web Workers: Pose and face tracking run off-thread via MediaPipeVisionWorker, preventing main-thread drops.
• Pose Landmarks: Positions body skeletons using real-world 3D metric landmarks with customizable confidence thresholds (minJointVisibility, minJointPresence).
• Depth & Occlusion: matchDepthView disabled by default in OcclusionOptions for direct camera pose alignment; duplicate second-eye and unchanged simulator depth readbacks are skipped.
• Simulation Clock & Physics: getElapsedTime('simulation' | 'render') freezes simulation time during pause; manual frame stepping accumulates fixed physics steps accurately.

Add-ons & ModelViewer

• objects3d: Detects 3D oriented bounding boxes using 2D masks (MediaPipe/SlimSAM), depth projection, and multi-view temporal clustering.
• virtualkeyboard: Rebuilt with retained UI elements, support for full 3D spatial transforms, and a new standalone sample.
• ModelViewer: Rebuilt on the unified interaction and UI pipeline, supporting .gltf, .glb, .ply, .spz, .splat, and .ksplat with normalized manipulation proxies and animation control.

Spatial Anchors (Quest 3 & Simulator)

• Added xb.world.anchors (AnchorManager) and AnchoredObjects for anchoring and persisting scene objects across sessions.
• Includes capability probing (AnchorCapability), storage management (LocalStorageAnchorStore), and desktop fallback (SimulatorAnchor).
• Note: Persistence is currently supported on Meta Quest 3 (and in the desktop simulator); Android XR persistence will become available once supported by the Chrome WebXR implementation.

---

Breaking Changes & Migration

Removed / Changed	Replacement
uiblocks entry & legacy UI (View, Panel, SpatialPanel, TextView, etc.)	Import UICard, UIOverlay, UIButton, UIText directly from xrblocks.
Options.enableUI(), Options.uikit, UIKitOptions	UI is initialized automatically on demand when adding UI elements to the scene.
DragManager, DragMode, Script.ux, draggable flags	Configure object.xb.manipulation = { translate: true, rotate: true, scale: true }.
ignoreReticleRaycast, custom raycasters	Use object.xb.pointerEvents, object.xb.reticleMode, and xb.user raycast queries.
SelectEvent.target was the controller	event.target is now the logical target object. Physical input is on event.source.controller.
Callback returning true to stop propagation	Call event.stopPropagation().
Synchronous Core.dispose()	await xb.core.dispose() is asynchronous and terminal.
Single-file bundle distribution	Deploy the full build/ directory including lazy build/internal/ chunks.

👉 For step-by-step instructions, see the complete Migrating to v0.20.0 Guide.

What's Changed

• Bump actions/setup-node from 6.5.0 to 7.0.0 in the ci-actions group across 1 directory by @dependabot[bot] in #472
• Add XREAL Aura by @salmanmkc in #488
• feat(addons/objects3d): reusable 3D object detection (2D detect + depth into oriented boxes) by @salmanmkc in #417
• perf(world/humans): run MediaPipe PoseLandmarker in a web worker by @salmanmkc in #489
• Disable matchDepthView by default and update occlusion by @dli7319 in #491
• fix(virtualkeyboard): honour the keyboard transform by @salmanmkc in #495
• place the pose skeleton from MediaPipe's metric landmarks by @salmanmkc in #503
• skip the second eye's depth readback by @salmanmkc in #507
• clear the physics interval and free the rapier world on dispose by @salmanmkc in #506
• skip the simulator depth readback when nothing has changed by @salmanmkc in #505
• fix(evals): run the prototype sweeps on Windows by @salmanmkc in #510
• feat(world/humans): let callers reject pose joints the model only guessed by @salmanmkc in #504
• fix(evals): stop comments satisfying an expected API by @salmanmkc in #509
• feat(mcp): add a server exposing the xb-* skills and API surface by @salmanmkc in #508
• feat(evals): cover head gestures, lipsync, context and agent hands by @salmanmkc in #514
• build(deps): bump js-yaml from 4.3.0 to 4.3.1 in /docs in the npm_and_yarn group across 1 directory by @dependabot[bot] in #511
• event.stopPropagation() on callbacks by @whuang37 in #512
• Realigned UI Rendering Backend by @whuang37 in #513
• v0.20 by @whuang37 in #517
• fixed packagelock dependencies by @whuang37 in #518
• Update brace-expansion and nanoid by @dli7319 in #519
• Migrated Virtual Keyboard by @whuang37 in #520
• feat(world/anchors): persistent spatial anchors by @salmanmkc in #494
• Realigned Physics Stepping When Pause by @whuang37 in #521
• created migration guide by @whuang37 in #522

Full Changelog: v0.19.0...v0.20.0

v0.19.0: 3D Scene Editor, Google GNM Avatars, Simulator Physics & Gemini 3.6 Flash

XR Blocks v0.19.0 Release Notes

We are excited to announce XR Blocks v0.19.0, which introduces the interactive 3D Scene Editor addon with transform gizmos and inspector panels, the Google GNM showcase demo of parametric 3D human head avatars, declarative Simulator Scene Objects & Physics with Rapier colliders and detection bridging, a temporary PinchFilter workaround for premature Galaxy XR pinch detection, and upgraded AI defaults with Gemini 3.6 Flash.

---

Highlighted New Features

🎨 3D Scene Editor Addon (xrblocks/addons/editor)

A powerful, interactive 3D Scene Editor addon that enables developers and users to build, inspect, and modify XR scenes in both desktop browser and WebXR/VR environments:

• 3D Transform Gizmo (TransformGizmo): Intuitive 3D handles and visual manipulators for translating, rotating, and scaling objects directly in 3D space.
• Comprehensive Editor UI Panels:
  • HierarchyPanel: Inspect, search, and navigate the full scene graph hierarchy.
  • ModelPickerPanel: Browse, preview, and spawn 3D GLTF/GLB models directly into the active scene.
  • ScenePanel: Control global scene settings, environment manifests, and lighting parameters.
  • TransformInspectorPanel: Perform precise numeric edits to object position, rotation, and scale properties.
• Scene & Selection Management (SceneManager, SelectionManager): Integrated object lifecycle management, scene serialization, and multi-selection support.
• Undo/Redo Command History (CommandHistory): Full transactional action stack supporting undo and redo operations across transform and scene edits.
• Simulator Integration (SimulatorEditorMode): Dedicated desktop editor control mode for seamlessly editing scenes in the browser simulator.

👤 Google GNM Parametric Human Head Demo (demos/gnm)

An interactive XR Blocks showcase demo of Google's Generative aNthropometric Model (GNM) v3.0, the 3D statistical parametric human head model that powers Android XR's Likeness feature:

• Sub-millisecond Client Forward Engine (GNMModel.js): Ported full forward evaluation to client-side JavaScript, computing 17,821 vertices across 253 identity and 383 expression PCA components in real time.
• Live Semantic Sampling (SemanticSampler.js): Generates realistic head variations by sampling gender, ethnicity, and 20 facial expression classes (happy, surprise, pucker, etc.) entirely client-side.
• Synchronized DOM & Spatial UI (GNMSpatialUI.js, GNMControls.js): Seamlessly control facial expressions, gaze tracking, joint rotation sliders, and animation drivers from either desktop DOM controls or spatial VR panels.
• Inspection & Material Modes: Inspect avatars across six rendering modes: Studio, Clay, Normals, Wireframe, Landmarks, and Skeleton.
• Dynamic Asset Loading & State Persistence: Loads int8-quantized model weights (gnm_head_web.bin, gnm_samplers_web.bin) dynamically from a CDN, with built-in scene saving and loading support.

🏡 Simulator Scene Objects & Physics Architecture (src/simulator/scene/)

Extends the desktop simulator with declarative environment manifests and physics-aligned scene simulation:

• Declarative Environment Manifests (SimulatorEnvironmentManifest, SimulatorEnvironmentManager): Load predefined indoor environments (office.json, living-room.json, emulator-scene-dark.json, emulator-scene-v5.json) with semantic tags, furniture geometry, and collision shapes.
• Rapier3D Physics-Aligned Simulation (SimulatorPhysics, SimulatorObjects, SimulatorGeometry): Realistic physical collider representations for simulator scene objects.
• Unified Spatial Detection Bridge (SimulatorObjectDetectionSource): Automatically feeds simulator scene objects into ObjectDetector, MeshDetector, and PlaneDetector, enabling full desktop testing of spatial AI/XR detection pipelines without physical hardware.
• Showcase Demo: Includes demos/simulator_scene_objects/ demonstrating live interaction and spatial detection with simulator scene objects.

🤏 PinchFilter Galaxy XR Workaround (src/input/gestures/)

Introduces a temporary workaround utility to mitigate premature pinch detection caused by a system-level gesture tracking bug on Galaxy XR:

• Temporary OpenXR Pinch Threshold Workaround (PinchFilter): Introduces PinchFilter (src/input/gestures/PinchFilter.ts), a temporary filtering workaround for Galaxy XR devices. In OpenXR, pinchValue can be evaluated as either a float (0.0–1.0) or a boolean, where 1.0 indicates a full pinch. On Galaxy XR, a system-level bug caused the boolean pinch state (read by Chrome) to trigger prematurely at 0.7 instead of 1.0. PinchFilter corrects for this premature triggering.
• Mobile Touch Preservation: Ensures PinchFilter explicitly ignores and preserves mobile touch events, maintaining smooth touch interactions on mobile browsers and tablets without interference.

🤖 AI, Embodied Control & Performance Optimizations

• Gemini 3.6 Flash Default (src/ai/AI.ts): Updated the default fast AI model to gemini-3.6-flash for improved reasoning speed and multimodal accuracy.
• Minimal Thinking Budget for Live Object Detection: Optimized Gemini object detection calls to use minimal thinking budget (thinkingBudget: 0), significantly reducing latency and token consumption during real-time spatial detection.
• Simulation-Tick-Aligned Embodied Control: Decoupled agent embodied movements from browser rendering framerates by aligning motion to simulation ticks, preventing jitter and speed variance across devices.
• Simulator Poses in Embodied Control: Added support for using simulator poses within the embodied-control addon with clear selection precedence rules.
• AgentHands Ergonomics: Repositioned AgentHands spatial control panels closer to the user's neutral gaze angle for enhanced comfort.

---

Technical Upgrades, Refactors & Bug Fixes

• Lazy BVH Raycast Acceleration (src/input/FaceRecognizer.ts): Loaded three-mesh-bvh raycast acceleration structures lazily inside FaceRecognizer.init(), speeding up startup and reducing bundle memory footprint when face mesh raycasting is not immediately required.
• Compressed Texture & GLTF Loader Enhancements (src/utils/ModelLoader.ts): Added KTX2/KTX compressed texture loader support on GLTFLoader creation and ensured unique GLTFLoader instances per ModelLoader to prevent texture and extension state collisions.
• CI & Zizmor Security Hardening (.github/workflows/): Fixed security audit findings identified by zizmor across GitHub Actions workflows (build-branch.yml).
• Standardized HTML Headers: Updated HTML headers across demos and samples, standardizing titles and removing redundant cache-control metadata.
• Streamlined Demo Contribution & PR Templates: Added comprehensive demo contribution guidelines (demo_contributing.md), updated sample/demo READMEs, and introduced standardized templates for pull requests and new demos/samples.
• Dependency & Tooling Upgrades: Upgraded js-yaml (to 4.3.0), postcss, and npm/yarn group dependencies, along with ESLint rules and documentation updates.

---

What's Changed

• scene editor addon by @Peter-Pater in #433
• feat(demos/gnm): Added Google GNM demo of parametric avatars in XR Blocks. by @ruofeidu in #443
• keep the AgentHands controls closer to neutral gaze by @salmanmkc in #444
• Physics Aligned Simulator Scenes and Detection by @whuang37 in #441
• feat(input): add PinchFilter helper to prevent false hand pinches by @dli7319 in #445
• Bump brace-expansion from 1.1.15 to 1.1.16 in the npm_and_yarn group across 1 directory by @dependabot[bot] in #447
• Revert "Physics Aligned Simulator Scenes and Detection" by @ruofeidu in #449
• feat: add GNM Head sample, update documentation titles, and reorganiz… by @ruofeidu in #451
• Fixed Sim Manifest by @whuang37 in #450
• refactor: standardize HTML headers by updating titles and removing re… by @ruofeidu in #452
• GLTF Reuse Fix by @whuang37 in #453
• Change default flash model to Gemini 3.6 Flash. by @dli7319 in #454
• added ability to use simulator poses in embodied-control by @whuang37 in #455
• feat(GNM): Added saving / loading features for GNM by @ruofeidu in #458
• Bump js-yaml from 4.2.0 to 4.3.0 by @dependabot[bot] in #460
• Aligned Embodied Control to Simulation Clock by @whuang37 in #457
• New Skills by @whuang37 in #446
• Update dependencies by @dli7319 in #463
• Add demo contribution guidelines by @dli7319 in #465
• Add template for new demos and samples by @dli7319 in #466
• Streamline PR template and rename it to pull_request_template.md. by @dli7319 in #468
• Update sample and demo READMEs and move PR template to root .github by @dli7319 in #469
• Bump postcss from 8.5.15 to 8.5.23 in /templates/typescript in the npm_and_yarn group across 1 directory by @dependabot[bot] in https://github.com/google/xrblocks/...

v0.18.0: Agent Hands, Head Gestures, Scene Context & Simulator NavMesh

XR Blocks v0.18.0 Release Notes

We are excited to announce XR Blocks v0.18.0, which introduces the brand-new Agent Hands addon for expressive agent embodiment, a fully customizable Head Gestures recognition subsystem, a detailed Scene Context extraction system for LLM integration, and simulator NavMesh pathfinding and reach limit mechanics.

---

Highlighted New Features

🙌 Agent Hands Addon (xrblocks/addons/agenthands)

A powerful new addon that implements a pair of virtual hands, an interactive head presence, and speech synchronization. Designed to let agents express semantic and motion gestures when communicating with users in XR.

• Rigged Embodiment (AgentHand/AgentHands): A dual-hand rigging system allowing standalone posable hands that can point at arbitrary world coordinates (.pointAt(position)) or lock onto specific 3D bounds/meshes.
• Markup Gesture Parsing (AgentGestures): Automatically parses semantic motion markup tags (e.g. [wave], [beat], [size:big], [count:2], [point:...]) from the agent's textual response stream.
• Motion & Pose Offsets (AgentGestureAnimator): Layers custom procedural motion animations on top of primary aims and default hand poses.
• Gaze & Speech Sync (AgentSpeechConductor): Wires into speech output events via word-boundary callbacks to synchronize mouth visuals and hand gesture timings with verbal pacing.
• Glowing Presence (AgentHead): Introduces a floating glowing orb that represents the agent's head, pulsing dynamically while speaking and looking toward tracked objects.
• Background Scanning (AgentWorld): Periodically performs background updates of the surrounding scene, storing and grounding session-specific objects for pointing actions.

🙋 Head Gestures Recognition Subsystem (src/input/headGestures/)

A custom input subsystem that enables lightweight user input telemetry based on natural head movements:

• Nod & Shake Detection: Built-in heuristics for recognizing head nods and shakes.
• Configurable Sensitivity: Includes adjustable tuning parameters (timeouts, speed/velocity gates, angle thresholds) in HeadGestureRecognitionOptions to optimize detection and reject slow/accidental head shifts.
• Fine-Grained Telemetry: Includes direction vectors in events to provide orientation contexts of head gestures.

🔍 Scene Context Extraction System (src/context/)

Equips LLM agents with structured, high-fidelity semantic and visual representations of the local XR scene:

• Semantic Trees (SemanticTreeBuilder): Exposes the scene hierarchy as a serialized tree outlining coordinate positions, relationships, and bounding bounds of entities.
• Set-of-Mark overlay (SetOfMarkBuilder): Dynamically projects visual ID tags (markers) onto key objects for image-based visual-grounding tasks.
• Token-Efficient Serialization: Employs floating point rounding and automated optimization to keep context formats compact, conserving valuable context window tokens.

🗺️ Simulator NavMesh & Pathfinding

Extends the desktop simulator to support pathfinding and realistic physical limits:

• Simulator Navigation (SimulatorNavMesh): Implements pathfinding capabilities inside the simulator environment to guide navigation tasks across virtual spaces.
• Simulated Reach Limits: Adds hand range and angle clamping (reachLimit) to mock the physical biomechanical constraints of human arms.

---

Technical Upgrades, Refactors & Bug Fixes

• ThreeJS Memory Cleanup (ThreeDisposal.ts): Introduces clean resource disposal routines for meshes, geometries, textures, and materials to prevent memory leaks during repeated test cycles and session loads.
• Interactive Zooming: Adds mouse-wheel scaling support for model-viewer views inside desktop browser simulator frames.
• Memory Leak Mitigations: Fixed an issue where repeated screenshot capture promises could stack up, mitigating memory overheads in continuous automation runs.
• Optional World Sensors: Updated Core.ts configuration to register world-sensing sensors as optional, enabling cleaner mock runs on setups lacking hardware camera feeds.
• VRM 0.x Compatibility: Added backward compatibility flags to VRMAvatar loader scripts.
• Simplified Build Command: Introduces npm run build:sdk for accelerated rebuilds of core SDK bundles without recompiling heavy addons.

---

What's Changed

• agenthands:
  • Merge pull request #416 from @salmanmkc: Add agenthands addon, rigs, gestures, markup parser, speech conductor, orb head, and background scanner.
• head-gestures:
  • Merge pull request #438 from @whuang37: Heuristic nodding/shaking head gesture detection, sensitivity tuning options, and samples.
• context:
  • Merge pull request #435 & #431 from @whuang37: Scene context optimization, token rounding, SemanticTree building, and Set-of-Mark (SOM) markers.
• simulator & navigation:
  • Merge pull request #429 from @whuang37: NavMesh and simulated pathfinding support.
  • Merge pull request #426 from @whuang37: Simulated reach bounds, reach limits, and angle clamping.
  • Merge pull request #437 from @Yong-yuan-X: Scale ModelViewers using the mouse scroll wheel.
• core & optimization:
  • Merge pull request #432 from @whuang37: ThreeJS systematic resource disposal.
  • Merge pull request #430 from @whuang37: Dedicated SDK-only build scripts.
  • Merge pull request #425 from @gagarinyury: Requested world-sensing features declared optional.
  • Merge pull request #427 & #423 from @Peter-Pater: VRM 0.x backward compatibility for VRMAvatar.
• dependency updates:
  • Bumps to websocket-driver, js-yaml, and GitHub workflow dependencies.

New Contributors

• @gagarinyury made their first contribution in #425
• @Yong-yuan-X made their first contribution in #437

Full Changelog: v0.17.0...v0.18.0

v0.17.0: Remote/Embodied Control, Headless Testing & Advanced Perception

XR Blocks v0.17.0 Release Notes

We are excited to announce XR Blocks v0.17.0, which introduces powerful new addons for programmatic simulation control, remote automation, and headless testing, along with major upgrades to on-device MediaPipe perception (segmentation, face, and body pose tracking) and 3D object fitting.

Highlighted New Features

🤖 Programmatic Embodied User Control (xrblocks/addons/embodied-control)

A new addon that lets tests and demos programmatically drive an XR Blocks scene through simulated user interactions.

• Locomotion & Camera: Control camera movement (strafe, rise, forward) and camera rotation (pitch, yaw, roll).
• Simulated Hands: Move and rotate left/right hands, trigger hand poses, and execute WebXR-style select gestures.
• Deterministic Stepping: Supports applyControl() for real-time adjustments and step({durationMs}) for advancing the frame loop deterministically—perfect for repeatable testing.

🌐 Remote Automation Protocol (xrblocks/addons/remote-control)

A WebSocket-based protocol and relay server (npx xrblocks-remote-control) to control an XR Blocks page from external processes (e.g., Python, Node, or AI agents).

• Bidirectional RPC: Register scene-specific functions as tools that external clients can invoke.
• Built-In Tools: Exposes built-in tools out of the box to retrieve screenshots, teleport, look at targets, pinch/click, and get simulator/hand states.
• Multi-Session Support: Run multiple independent browser/agent runs Rent-Free over a single relay port using sessionId.
• Automation Preset: Use options.enableAutomationMode() or the ?xrAutomation=1 URL flag to auto-start the simulator in a clean headless-friendly configuration.

🧪 Headless Functional Testing (xrblocks/addons/testing)

A headless functional test runner (TestRunner) designed for writing sequential integration and unit tests in environments like Vitest and JSDOM.

• WebGL & WebAudio Stubbing: Automatically mocks WebGLRenderer capabilities and AudioContext nodes.
• Interaction Simulation: Simulate user locomotion, controller pointing, and pinch/click gestures on 3D meshes.
• Lifecycle Error Catching: Catches and reports any exceptions thrown during a script's lifecycle (init, update, etc.).
• Test Isolation: A simple destroy() teardown pattern prevents state leaking between test suites.

🧠 Advanced MediaPipe Perception: Segmentation, Faces & Body Poses

On-device perception has been fully overhauled and decoupled from object detection:

• World Segmentation (Segmenter): Computes per-pixel category masks (SegmentationMask) on the camera feed.
• Face Tracking (FaceRecognizer): Recovers 3D landmarks, rigid head transforms, and blendshape weights (supporting 478 landmarks).
• BVH-Accelerated Face Raycasting: Integrates three-mesh-bvh to perform hundreds of landmark-to-depth-mesh raycasts per frame without dropping frames.
• Body Pose Tracking (HumanRecognizer): Detects human body poses and projects 3D joint landmarks into world space.

📦 3D Object Detection & Fitting Upgrades

• Multi-Backend Detection: ObjectDetector now supports interchangeable backends (including Gemini and MediaPipe).
• Robust 3D Box Fitting: Improved yaw-fit OBB algorithms using per-category RANSAC plane fitting, snapping furniture to cardinal directions, and snapping tiny flat planes.
• Quest 3 RGB Camera Calibration: Built-in support for Meta Quest 3 RGB camera intrinsics, including a ~15° downward camera tilt offset.
• Visual Polish: Billboarded labels, occluded edge rendering (dual-pass), and size-consistent box name sprites.

🎭 VRM Avatar & Lipsync Integration

• VRM Avatars: Added support for loading VRM avatars, wired directly to Gemini and lipsync mouth animations.
• Performance Optimizations: Added options to disable raycasting against the avatar and the depth mesh to maintain high frame rates on standalone headsets.

---

🤖 AI Agent & Developer Skills System

📊 Skill Evaluation Harness (evals/)

A reproducible benchmarking suite designed to measure the quality of code generated by Gemini Canvas Gems with and without specific xb-* skills.

• Dual-Mode Testing: Runs tasks through Gemini models with and without skill files injected in the system prompt.
• Automated Scoring: Measures code generation quality using structural metrics (import_match, api_match, forbidden_clean) and syntax checking (parse_ok).
• Advanced Verification: Includes an LLM-as-a-judge scorer (rating task accomplishment and idiomatic xrblocks usage), a Playwright-based headless Chromium smoke tester to catch runtime errors, and ablation testing to isolate the impact of individual skill sections.

---

Technical Upgrades & Refactors

• Vite + TypeScript Template: Added a new modern typescript template under templates/typescript/ utilizing Vite.
• Renderer Adjustments: Allowed the renderer to be undefined before initialization, and unified the renderer getter to always return a WebGLRenderer.
• UI Cleanups: Removed deprecated onHoverOver and onHoverOut methods from View classes.

What's Changed

• Simulator: Add pointer lock mode. by @dli7319 in #354
• Simulator: Resolve mouse reticle lag in Simulator User Mode. by @dli7319 in #355
• Simulator: add Instructions button to settings panel. by @dli7319 in #356
• Gemini vrm lip by @Peter-Pater in #357
• evals: harness for the xb-* skills (first cut) by @salmanmkc in #351
• build(deps): bump the npm_and_yarn group across 2 directories with 1 update by @dependabot[bot] in #358
• feat: implement missing features in ui, added test coverage, and revise logs by @ruofeidu in #359
• demos: add diet helper XR demo with Gemini photo analysis by @JFGVL1187 in #332
• demos: add new features to dietHelper by @JFGVL1187 in #360
• Pose landmark detection by @samiriff in #361
• world: add FaceRecognizer + face_mirror demo by @salmanmkc in #364
• perf(world/faces): run MediaPipe FaceLandmarker in a web worker by @salmanmkc in #366
• perf(simulator): don't stack depth-readback requests in flight by @salmanmkc in #370
• perf(world/faces): BVH-accelerate per-landmark depth raycasts by @salmanmkc in #367
• perf(demos/face_mirror): merge wireframe Lines into one LineSegments by @salmanmkc in #368
• fix(demos/face_mirror): anchor spatial HUD blendshape bars left by @salmanmkc in #369
• perf(depth/occlusion): skip OcclusionPass setSize when dimensions unchanged by @salmanmkc in #371
• build(deps): bump the npm_and_yarn group across 2 directories with 2 updates by @dependabot[bot] in #375
• build(deps): bump the npm_and_yarn group across 1 directory with 2 updates by @dependabot[bot] in #376
• refactor: simplify FaceMirror orchestrator and modularize face landmarker demo components by @dli7319 in #365
• perf(utils): applyBVH helper for opting scenes into accelerated raycasts by @salmanmkc in #372
• build: include src in npm uploads by @dli7319 in #380
• docs: add typescript template and guide for npm usage by @dli7319 in #381
• build(deps): bump esbuild from 0.25.12 to removed in /templates/typescript in the npm_and_yarn group across 1 directory by @dependabot[bot] in #382
• objects_3d: single-press 3D object detection demo by @salmanmkc in #363
• docs: fill in missing capabilities and addon manuals, reorganize sidebar by @dli7319 in #383
• KP to Hand Rotations by @whuang37 in #384
• chore: remove temporal-polyfill and lorem-ipsum by @dli7319 in #385
• objects_3d: use the xb BVH helper instead of patching THREE inline by @salmanmkc in #378
• objects_3d: make the control panel draggable by @salmanmkc in #379
• fix face_mirror: camera-not-ready spam + stale depth-mesh cache by @salmanmkc in #386
• Embodied Control + Remote Control by @whuang37 in #388
• docs: add Face Mirror sample page by @salmanmkc in #390
• Fix camera cycling when permission not yet granted by @nsalminen in #392
• docs: add 3D Object Boxes sample page by @salmanmkc in #389
• Avatar performance fix by @Peter-Pater in #393
• Semantic embodied actions by @whuang37 in #394
• add magic_window: segment people out of the camera onto a swappable backdrop by @salmanmkc in #387
• JSDOM Headless Testing by @whuang37 in #399
• Balloon Pop updates, UI tweaks by @jamiesonpepper in https...

v0.16.0: Audio Lipsync, FK Hands & Autoplacement

XR Blocks v0.16.0 Release Notes

We are excited to announce XR Blocks v0.16.0, which brings major new subsystems, visual enhancements, and improvements to XR Blocks developer and agent workflows!

Highlighted New Features

🗣️ Audio-Driven Lipsync Addon & Stylized Face Decals

This release introduces the lipsync addon, bringing low-latency, audio-driven mouth animations to user avatars.

• Heuristic Vowel Formants: Evaluates incoming MediaStream frequencies frame-by-frame using pure digital signal processing (DSP) to map speech into 6 core viseme weights (jawOpen, aa, oo, oh, ee, consonant) with frame-rate independent smoothing. Requires no ML model downloads or separate runtimes!
• Dynamic Blinking Eyes: Introduces xb.StylizedFace, a lightweight core visual component that draws dynamic blinking eyes and mouth decals to a canvas texture, rendering flush against head meshes.
• Netblocks Multiplayer Integration: RemoteUserAvatar now attaches a face decal out-of-the-box. Developers can map remote audio streams directly to each peer's avatar face, enabling avatars that visually react when speaking in multiplayer sessions.

👐 Biomechanical Simulated Hands & Forward Kinematics (FK)

The desktop simulator has received a massive mechanical overhaul for simulated hands:

• Forward Kinematics (FK): Hand joints and pose presets are now resolved using a biomechanical Euler angle rotation model (HandPoseFK) rather than translation offsets, resulting in more natural-looking fingers.
• Biomechanical Constraints: Includes enforcement of real joint constraints (e.g. thumb MCP/IP flexion-extension limits) to prevent impossible hand bends in simulator tests.
• Manual Sim Controls & Presets: Adds a dedicated hand pose visualizer/test tool, new hand pose presets (neutral flat hand, points, pinching, thumbs-up/down, victory, rock-on), and support for rotational angles in presets.

✌️ Refactored Poses and Gestures API

The webcam and hand-tracking system has been fully refactored to decouple pose estimation from gesture detection:

• Estimator/Detector Pipeline: The tracking loop is now divided into two clear stages: Pose Estimator (MediaPipe, TensorFlow, or WebXR) $\rightarrow$ Gesture Detector.
• Extensible Custom Gestures: Simpler, cleaner API for registering custom hand gestures.
• Refactored Heuristics: Improved built-in gesture detectors that work seamlessly out of the box.

🖼️ Horizontal Autoplacement API

A new placeOnHorizontalSurface API makes placing 3D objects onto detected real-world surfaces trivial.

• Smart Scoring: Automatically finds, filters (ignoring ceiling/upside-down planes), and scores candidate surfaces (tables, desks, counters, floors) based on distance, user look direction alignment, and height preference.
• Validation & Collision Resolution: Validates object bounds to ensure they fit correctly on the target surface without overlapping walls, controllers, or other obstacles, yielding frame-by-frame (waitFrame) until a safe spot is found.

⌨️ Virtual Keyboard Refresh

The virtual keyboard addon has been refactored for better parity with latest types:

• Enhanced Visual Depth: Polished UI layout logic and rendering depths.
• Linked Output Box: Added out-of-the-box support for piping virtual keyboard input directly into target text display fields.

🕶️ Glasses UI & UIBlocks Enhancements

• Emoji Support: Full emoji rendering support added directly into Glasses UI cards.
• Text wrapping & Formatting: Fixed issues preventing leading spaces on wrapped lines and added standard newline character support in TextWithEmoji.
• Model Resource Management: Added GlassesModelManager to ease loading and rendering of eyewear models.
• Aesthetic Polish: Auto-hiding text overflow in Glasses UI cards and corrected action button icon sizing.

---

Technical Upgrades & Refactors

• TypeScript 6.0: Core library and documentation upgraded to TypeScript 6.0, including built-in type definitions for DOM Speech Recognition.
• Gemini Interactions API: Upgraded @google/genai to v2.7.0 to support the Gemini interactions endpoints, widened AI.query input support, and resolved dispatch issues in AI.generate.
• Stable Frame Timing: Migrated internal frame duration loops from THREE.Clock to THREE.Timer.
• UIKit Auto-Sorting: Added configuration flags to automate translucent depth sorting (reversePainterSortStable) and local clipping for @pmndrs/uikit panels.
• three.js Upgrade: Peer dependency bumped to ^0.184.0.
• Depth downsampling: Support downsampled depth mesh geometry in ObjectDetector to speed up collision queries.

---

🤖 AI Agent & Developer Skills System

To assist developer onboarding and AI agent coding partners, this release packages:

• In-Tree Subsystem Documentation: Added a structured index of agent-targeted guides under skills/ (xb-core, xb-ui, xb-hands, xb-lipsync, xb-physics, xb-simulator, etc.).
• Alignment Specifications: Core specification files (GEMINI.md, AGENTS.md, CONTEXT.md) have been added to help AI coding agents build applications with consistent APIs and correct conventions.

---

What's Changed

• refactor: migrate THREE.Clock to THREE.Timer by @dli7319 in #303
• Upgrade @sparkjsdev/spark to 2.1.0 to fix shader issue by @nsalminen in #304
• Add dependency caching, test runs, and precise path triggers in workflows by @nsalminen in #305
• Upgrade actions/cache to v5 in workflows by @nsalminen in #306
• Document WebXR mesh detection browser & device requirements by @JFGVL1187 in #307
• Add support for downsampled depth mesh geometry in ObjectDetector by @nsalminen in #308
• Upgrade to TypeScript 6.0 by @dli7319 in #309
• Remove path triggers for PRs. by @dli7319 in #312
• Update docs typescript to 6.0 by @dli7319 in #311
• Autoplace on horizontal surface API. by @dli7319 in #310
• Tune model viewer sample. by @dli7319 in #313
• Fix big buck bunny url in samples/ui/ by @dli7319 in #315
• Model viewer scene: Fix globe sizing in panel. by @dli7319 in #314
• Add emoji support to Glasses UI. by @dli7319 in #318
• samples/ui/: Change to a veo-generated duck swimming video. by @dli7319 in #320
• Update @google/genai to 2.7.0 to support gemini interactions api by @dli7319 in #319
• CC0 animation and use cdn paths by @Peter-Pater in #316
• Forward Kinematics on Simulation Hands for AI Gen by @whuang37 in #321
• Add uikit options for automatic @pmndrs/uikit renderer setup by @dli7319 in #317
• update demo to use three.js v184 by @Peter-Pater in #322
• build(deps-dev): bump vitest from 3.2.4 to 4.1.0 in the npm_and_yarn group across 1 directory by @dependabot[bot] in #324
• Add GlassesModelManager.ts. by @dli7319 in #325
• Adding the vrm avatar demo to the sample list on the github io page by @Peter-Pater in #326
• Sim Hand Pose Migration to Angular Representation by @whuang37 in #323
• docs: fix Templates sidebar index item and rename Ai-Live to AI-Live by @ruofeidu in #327
• Add newlines support in TextWithEmoji by @dli7319 in #328
• fix(uiblocks): prevent leading spaces on wrapped lines in TextWithEmoji by @dli7319 in #330
• Glasses UI: Fix action button icon sizing. by @dli7319 in #329
• Glasses UI: Hide text overflow in Card. by @dli7319 in #331
• Refactor XR Keyboard layout and UI rendering by @jolicaD in #156
• Add a skills file for uiblocks. by @qxziuan in #336
• Fix interactions in Netblocks sample. by @dli7319 in #335
• Restore .vscode/.gitignore and .vscode/extensions.json by @dli7319 in #334
• Add lipsync addon: audio-driven mouth animation for avatars by @salmanmkc in #338
• Remove undefined xb.xrLightingOptions. by @dli7319 in #341
• lipsync sample on the docs site by @salmanmkc in #342
• uiblocks: use xrblocks@build CDN like the rest of the docs by @salmanmkc in #344
• MaterialSymbolsView: read viewBox via getAttribute by @salmanmkc in #346
• fix AI.generate dispatch + isAvailable check by @salmanmkc in #345
• widen AI.query input to accept GeminiQueryInput by @salmanmkc in #343
• skills: Added top-level design for GEMINI, CONTEXT, AGENTS, SKILLS. by @ruofeidu in #339
• Restore bundlephobia badge by @dli7319 in https://github...

v0.15.0: Netblocks multiplayer, One dollar Gesture Recognizer, Depth on mobile!

What's Changed

• tflite files cleanup under samples/ by @zhongyi-zhou in #290
• Depth support for mobile. by @dli7319 in #291
• Add netblocks addon for multiplayer XR sessions by @salmanmkc in #284
• Mediapipe Sound Detection - PR Comments Follow-up by @samiriff in #292
• Catch exceptions in script lifecycle methods. by @dli7319 in #293
• Refactor prototype .bind(this) wrappers to arrow functions. by @dli7319 in #294
• Update default gemini models. by @dli7319 in #295
• Fix depth mesh on Quest 3. by @dli7319 in #296
• build(deps): bump webpack-dev-server from 5.2.2 to 5.2.4 in /docs in the npm_and_yarn group across 1 directory by @dependabot[bot] in #297
• Cache depthDataFormat in Depth and move GPU depth conversion logic to GPUDepthConverter by @dli7319 in #299
• One dollar gesture estimator 2 by @samiriff in #298
• Update to threejs r184 by @dli7319 in #300
• build(deps): bump the npm_and_yarn group across 2 directories with 2 updates by @dependabot[bot] in #301
• Bump to 0.15.0 by @dli7319 in #302

Full Changelog: v0.14.1...v0.15.0

v0.14.1: Bugfix patch for Gemini Canvas XR Blocks Gem

What's Changed

• docs: add Language Detector sample page by @salmanmkc in #279
• Use more configs in the uiblocks sample. by @qxziuan in #283
• fix(simulator): ignore gamepad input when tab isn't focused, also switching LT/RT default to match Q/E movement controls for controller and desktop parity by @salmanmkc in #285
• Attach onWindowResize listener earlier. by @dli7319 in #286
• Bump to 0.14.1 by @dli7319 in #287

Full Changelog: v0.14.0...v0.14.1

v0.14.0: Mediapipe Sound Detection, Portal Demo, Simulator Gamepad Support

What's Changed

• chore(deps-dev): bump postcss from 8.5.8 to 8.5.12 in the npm_and_yarn group across 1 directory by @dependabot[bot] in #256
• Disable simulator instructions by default. by @dli7319 in #257
• remove the local tflite files and use external CDN to link the tflite model by @zhongyi-zhou in #258
• Add shader portals demo by @salmanmkc in #261
• Mediapipe Sound Detection by @samiriff in #259
• VRM Avatar — Point-to-Walk Companion Demo by @Peter-Pater in #239
• chore(deps): bump the npm_and_yarn group across 2 directories with 2 updates by @dependabot[bot] in #264
• Fix/canvas text align fontsize by @salmanmkc in #203
• Walk-in portal immersives by @salmanmkc in #265
• Add gamepad support to simulator by @salmanmkc in #262
• Remove duplicate rollup build from serve script by @nsalminen in #269
• Add Portal Gallery to docs samples by @salmanmkc in #267
• chore(deps): bump @babel/plugin-transform-modules-systemjs from 7.28.5 to 7.29.4 in /docs in the npm_and_yarn group across 1 directory by @dependabot[bot] in #266
• Simulator: Disable mode toggle and mode indicator by default. by @dli7319 in #270
• Create simulator settings panel. by @dli7319 in #271
• Update @sparkjsdev/spark to v2.0.0 and optimize XR rendering by @nsalminen in #273
• Update docusaurus to 3.10.1 by @dli7319 in #272
• language detector demo - uses gemini live for the audio stream + mediapipe's language detector by @salmanmkc in #274
• Make enableCamera also request microphone permission. by @dli7319 in #275
• docs: fix spelling errors in README documentation by @dli7319 in #276
• Move microphone permission request from enableCamera() to gemini_icebreakers by @dli7319 in #277
• Add a UI Blocks sample to the Spatial UI section of the XR Blocks samples doc. by @qxziuan in #278
• Update default Gemini flash and live model versions by @dli7319 in #281
• Bump to version 0.14.0 by @dli7319 in #282

New Contributors

• @Peter-Pater made their first contribution in #239

Full Changelog: v0.13.0...v0.14.0

v0.13.0: Mediapipe object detection!

What's Changed

• Standardize file naming conventions and directory structures across demos and samples by @dli7319 in #240
• chore(deps-dev): bump vite from 7.3.1 to 7.3.2 in the npm_and_yarn group across 1 directory by @dependabot[bot] in #241
• Add option to disable the simulator camera. by @dli7319 in #243
• Glasses UI: Hide weather until we have a result. by @dli7319 in #244
• optimize the performance (fps) of mesh detector in xrblocks by @JFGVL1187 in #144
• chore(deps): bump lodash from 4.17.23 to 4.18.1 in /docs in the npm_and_yarn group across 1 directory by @dependabot[bot] in #242
• Mediapipe object detection by @samiriff in #245
• chore(deps): bump the npm_and_yarn group across 2 directories with 1 update by @dependabot[bot] in #247
• Fix some typedoc warnings. by @dli7319 in #250
• Fix XR Poet. by @dli7319 in #251
• Prettier ignore build/ and docs/docs/api/ by @dli7319 in #249
• Add Torus loading animation to XR Poet by @dli7319 in #252
• Make ToolSchema.items have type ToolSchema by @dli7319 in #253
• aisimulator: Update gemini live model and pipe text inputs. by @dli7319 in #254
• Bump to 0.13.0 by @dli7319 in #255

New Contributors

• @samiriff made their first contribution in #245

Full Changelog: v0.12.0...v0.13.0

v0.12.0: UI Blocks library and Glasses UI library

What's Changed

• Fix/camera play error handling + test harness setup by @salmanmkc in #202
• chore: add Dependabot configuration by @salmanmkc in #206
• Revert "chore: add Dependabot configuration" by @dli7319 in #223
• Add uiblocks addon and an xrblocks integrated sample. by @qxziuan in #199
• Bump the npm_and_yarn group across 2 directories with 3 updates by @dependabot[bot] in #225
• Add Dependabot configuration for GitHub Actions by @salmanmkc in #224
• Support TypeScript workflow with gHacks demos/animalattack. by @duckladydinh in #200
• fix: implement graceful fallback and timeout for WebXR camera access by @salmanmkc in #227
• Update camera state message for clarity, could be browser/device by @salmanmkc in #229
• Make OptionsUtil deepMerge always clone the object. by @dli7319 in #228
• Bump node-forge from 1.3.2 to 1.4.0 in /docs in the npm_and_yarn group across 1 directory by @dependabot[bot] in #230
• Feat/depth mesh normview transform by @salmanmkc in #204
• Add initial glasses ui library and sample. by @dli7319 in #231
• Added CITATION.cff and updated Vibe Coding XR links. by @ruofeidu in #232
• Make /docs/templates/ point to /docs/templates/Basic/ by @dli7319 in #233
• Add reticle defaultDistance fallback for no-intersection case by @JFGVL1187 in #235
• Adopt demos/drone to the new TypeScript support. by @duckladydinh in #234
• chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates by @dependabot[bot] in #236
• Bump to version 0.12.0 by @dli7319 in #237

New Contributors

• @duckladydinh made their first contribution in #200

Full Changelog: v0.11.0...v0.12.0