real machine

[Mayunsheng-1]

Hello author, long time no see. Some time ago, I was trying to deploy the VR remote control in the simulation to the real machine through sim to real. But I failed for some reasons. I would like to ask you how you realize the VR remote control of the real machine. Can you teach me? In addition, I want to know whether the remote control screen in VR can be synchronized to the computer.
I am very sorry to bother you. I wish you academic success and continuous promotion.

[ian-chuang]

Hi,

No problem, just share any errors that you come across. I would probably first confirm that you can indeed stream real video (from ZED mini) to the VR headset. If you analyze real_env.py and ZEDImageRecorder in image_recorders.py hopefully you can get a sense of how I do this.

For how VR control is realized I would really analyze this script here to get a sense of how it works:

av-aloha/data_collection_scripts/real_env.py

Line 389 in 65da520

headset_action, feedback = headset_control.run(

Basically I have created an environment where the middle arm is controlled by sending a pose (position + quaternion). I retreive the headset pose from WebRTCHeadset and then I do some preprocessing on the pose (transforming it to robot's frame of reference) with HeadsetControl, and this pose is then passed to RealEnv which does the inverse kinematics to calculate robot joints which is sent to InterbotixSDK to control arms.

Unfortunately I'm not sure how to stream the exact VR view from headset to computer. I'm only aware of just recording the VR view using the Meta Quest recording features. At the very least you can view the camera stream sent to VR here:

av-aloha/data_collection_scripts/real_env.py

Line 167 in 65da520

'zed_cam_left': zed_image[:, :1280, :],

Also I recommend running the "fake" simulated robot arms in ROS before trying the real arm by running this command instead:

roslaunch guided_vision 3arms_teleop.launch use_sim:=true use_rviz:=true