Synchronizing the video and gaze data on tobii glasses

[oriki101]

Hello, thanks for sharing the codes for controlling tobii glasses!
I am trying to get a map between videos and gaze data online, but I am getting a map with a delayed image and a real-time gaze data.
So I would be happy if you could tell me the way to get an online map between videos and gaze data correctly.
Thanks in advance!

[ddetommaso]

Hi @Kazuaki9817

please can you post your script or refer to one of the examples? Are you using WiFi or eth connection? How is the live view with VLC, is it still delayed?

[FranzAlbers]

We are experiencing the same issue using either WiFi or Ethernet connection. It can be reproduced with the live_scene_and_gaze.py example.
Whenever you fixate one point while moving your head to one side, the gaze point in the Livestream begins to move in the opposite direction for about 0.5-1 second, instead of staying at this point. This does not happen during a Livestream in the Glasses Controller Software from Tobii.

Seemingly, gaze data and video data are not synchronized correctly. Gaze data appears to arrive a little bit earlier than video data, which causes the gaze point in the Livestream to drift during head movements.

[ddetommaso]

Hi @FranzAlbers ,

I didn't experience yet this issue. I only have some freeze sometime using wifi connection.
Which operating system are you using?

[FranzAlbers]

Hey @ddetommaso,

we originally used this software and experienced the issue in our driving simulator with Windows, but I can reproduce it also on my workstation using ubuntu (with both Python 2.7 and 3.6). Besides, thanks for publishing this useful package!

I believe the issue is actually rather part of the live_scene_and_gaze.py in the Examples, where the offset value (in seconds) is compared to the frame_duration (which is given in milliseconds) which results in incorrectly displayed gaze data:

frame_duration = 1000.0/float(video_freq) #frame duration in ms
...
offset = data_gp['ts']/1000000.0 - data_pts['ts']/1000000.0
if offset > 0.0 and offset <= frame_duration:`
    cv2.circle(frame,(int(data_gp['gp'][0]*width),int(data_gp['gp'][1]*height)), 30, (0,0,255), 2)

The gaze data is therefore currently (almost) always just added to the current frame, but not necessarily to the correct frame.

[ddetommaso]

Hi @FranzAlbers,

thank you for the feedback.
What you're saying is correct. At the moment is not clear how to synch video and gaze in the live view using openCV. So I cannot close the issue right now.

Please let me know if you guys find a solution for that!

[minhanli]

I think you might need to create a queue to buffer video and gaze data, and then utilize the offset to sync them as the example code provided by Tobii does. Just a thought, still trying to figure out how to realize.

[ddetommaso]

Hi @minhanli ,

I did check the example provided in the API but I am still unable to make it works with Opencv, cause they're using gstreamer. I am not able to extract the timestamp of the mpeg flow using OpenCV, so even if we have buffers of video and gaze data, I think they're pretty useless without both timestamps.

[minhanli]

Hi @ddetommaso,

I read the manual, and it says something about synchronization like below:

Did you try this? It seems like we only need to know the pts of the video, instead if ts.

[ddetommaso]

Hi @minhanli

I was referring to this documentation just the comment above.
I do not have the ts of the videostream. With OpenCv it seems I can access the videostream only frame by frame.

[minhanli]

@ddetommaso
You are right. We seem to have no way to access the timestamp info of the video thru the OpenCv, as the OpenCv discarded them way before. I'm thinking of using other packages to decode the video while keeping the timestamp, such as pyav. Not using gstreamer, because it is a hassle to deal with the installation.

[FranzAlbers]

By using pyav instead of opencv for grabbing the video, the delay is almost not perceivable. One example with pyav can be found in the following code:

import cv2
import av
import numpy as np
from tobiiglassesctrl import TobiiGlassesController

tobiiglasses = TobiiGlassesController(video_scene=True)
ipv4_address = tobiiglasses.get_address()
tobiiglasses.start_streaming()

rtsp_url = "rtsp://%s:8554/live/scene" % ipv4_address
container = av.open(rtsp_url, options={'rtsp_transport': 'tcp'})
stream = container.streams.video[0]

for frame in container.decode(stream):
    data_gp = tobiiglasses.get_data()['gp']
    data_pts = tobiiglasses.get_data()['pts']
    frame_cv = frame.to_ndarray(format='bgr24')

    if data_gp['ts'] > 0 and data_pts['ts'] > 0:
        # offset = data_gp['ts'] / 1000.0 - data_pts['ts'] / 1000.0  # in milliseconds
        # print('Frame_pts = %f' % float(frame.pts))
        # print('Frame_time = %f' % float(frame.time))
        # print('Data_pts = %f' % float(data_pts['pts']))
        # print('Offset = %f' % float(offset))

        # Overlay gazepoint
        height, width = frame_cv.shape[:2]
        cv2.circle(frame_cv, (int(data_gp['gp'][0] * width), int(data_gp['gp'][1] * height)), 20, (0, 0, 255), 6)
    # Display Stream
    cv2.imshow("Livestream", frame_cv)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cv2.destroyAllWindows()
tobiiglasses.stop_streaming()
tobiiglasses.close()

However, this is of course still not correctly synchronized. The better way would probably be to add something like the BufferSync Class from the Tobii Pro Glasses API Examples (video_with_gaze.py) to the TobiiGlassesPyController, buffer gaze data and return the synchronized gaze data based on the video pts we get from pyav.

[ddetommaso]

Hi @FranzAlbers,

thank you for your work! It's definitively worth to modify the controller with the BufferSync as you suggested. I'll work on it in the next days.

Stay tuned!

[HansenLars]

Hi, did you find a solution to that? I'm currently working on the same thing. I have tried with Gstreamer but couldn't even get the pipeline to work.

[ddetommaso]

Hi @HansenLars

unfortunately I don't have a solution right now.
I think the issue might be solved using OpenCV grabber. I saw that it is actually possible to access timestamp of a current frame (CAP_PROP_POS_MSEC in cv::VideoCaptureProperties) https://docs.opencv.org/4.5.0/d4/d15/group__videoio__flags__base.html