Skip to content

Conversation

@ooctipus
Copy link
Collaborator

Description

Introducing dexsuite with support for warp rendering, single camera version implemented.
merging this PR requires #4416 merged first

  • New feature (non-breaking change which adds functionality)

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions bot added the asset New asset feature or request label Jan 19, 2026
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Jan 19, 2026

Greptile Summary

  • Introduces the DexSuite manipulation task suite with warp rendering support featuring Kuka-Allegro hand environments for object lifting and reorientation tasks
  • Standardizes quaternion representation from (w,x,y,z) to (x,y,z,w) format across the codebase as part of issue [NEWTON] Fixes more xyzw issues in core #4416
  • Adds comprehensive vision-based reinforcement learning capabilities with CNN-based actor-critic networks and multi-camera observation processing

Important Files Changed

Filename Overview
source/isaaclab_newton/isaaclab_newton/assets/rigid_object/rigid_object_data.py Newton physics RigidObjectData implementation with logical errors in velocity kernels that could cause runtime failures
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/dexsuite/config/kuka_allegro/dexsuite_kuka_allegro_vision_env_cfg.py Vision environment config with type annotation mismatches and attribute deletion issues in post_init
source/isaaclab/isaaclab/cloner/cloner_utils.py Major refactor of Newton physics replication from single-world to per-environment approach affecting simulation architecture
source/isaaclab/isaaclab/sim/prims/xform_prim.py Quaternion convention change and return type changes from numpy to tensors that introduce breaking changes
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/dexsuite/adr_curriculum.py ADR curriculum with potential configuration error where noise max value incorrectly set to negative

Confidence score: 3/5

  • This PR introduces significant new functionality but contains several technical issues that require careful review
  • Score lowered due to logical errors in Newton physics implementation, type annotation mismatches, and potential breaking changes from quaternion format standardization
  • Pay close attention to the rigid object data implementation, vision environment configurations, and any code depending on quaternion ordering

Sequence Diagram

sequenceDiagram
    participant User
    participant InteractiveScene
    participant TiledCamera
    participant NewtonWarpRenderer
    participant NewtonManager
    participant ObservationManager

    User->>InteractiveScene: __init__(cfg)
    InteractiveScene->>InteractiveScene: _add_entities_from_cfg()
    InteractiveScene->>TiledCamera: __init__(cfg)
    TiledCamera->>TiledCamera: _initialize_impl()
    TiledCamera->>NewtonWarpRenderer: __init__(renderer_cfg)
    TiledCamera->>NewtonWarpRenderer: initialize()
    InteractiveScene->>InteractiveScene: clone_environments()
    InteractiveScene->>NewtonManager: newton_replicate()
    User->>InteractiveScene: reset(env_ids)
    InteractiveScene->>TiledCamera: reset(env_ids)
    TiledCamera->>NewtonWarpRenderer: reset()
    User->>ObservationManager: compute_group()
    ObservationManager->>TiledCamera: update()
    TiledCamera->>TiledCamera: _update_buffers_impl()
    TiledCamera->>TiledCamera: _update_poses()
    TiledCamera->>NewtonWarpRenderer: render(pos, quat, intrinsics)
    NewtonWarpRenderer-->>TiledCamera: output_buffer
    TiledCamera->>TiledCamera: wp.to_torch(output_buffer)
    TiledCamera-->>ObservationManager: camera.data.output
    ObservationManager-->>User: observations
Loading

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (3)

  1. source/isaaclab/isaaclab/sim/_impl/newton_manager.py, line 378 (link)

    style: The parameter prune_noncolliding is still in the function signature but is no longer used

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

  2. source/isaaclab_newton/isaaclab_newton/assets/rigid_object/rigid_object_data.py, line 719-728 (link)

    logic: This creates a quaternion array but the property should return linear velocity. The wrong kernel split_transform_array_to_quaternion_array is being used on pose data instead of splitting velocity data.

  3. source/isaaclab_newton/isaaclab_newton/assets/rigid_object/rigid_object_data.py, line 337-351 (link)

    logic: This property claims to return center of mass velocity but uses _body_com_pose_w.timestamp for cache invalidation and applies transform operations instead of returning the velocity binding _sim_bind_body_com_vel_w.

41 files reviewed, 32 comments

Edit Code Review Agent Settings | Greptile

@ooctipus ooctipus changed the title Newton/dexsuite warp rendering [NEWTON] support dextrah environments with warp rendering Jan 19, 2026
@ooctipus ooctipus force-pushed the newton/dexsuite_warp_rendering branch from c4e2cbb to 890de08 Compare January 19, 2026 12:33
@ooctipus
Copy link
Collaborator Author

don't merge this one, there is big asset issue, this pr is used for collaboration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

asset New asset feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants