Poor tracking of the cheeks and tongue

[unreal-2222]

I followed all the steps, but the tongue and cheeks are not being tracked. Also, in FoxyFace’s MediaPipe preview mode, there are no points on the tongue. As I understand it, the Babble model is responsible for this, but updating it doesn’t make any difference — the tongue and cheeks still aren’t tracked at all.

[Jeka8833]

Does the Babble model show an FPS greater than 0?
When I talk about FPS, this line can be found here: https://foxyface.jeka8833.pp.ua/docs/FoxyFace/ui/babble/Babble-Settings#what-is-fps-in-the-babble-category

[unreal-2222]

Does the Babble model show an FPS greater than 0? When I talk about FPS, this line can be found here: https://foxyface.jeka8833.pp.ua/docs/FoxyFace/ui/babble/Babble-Settings#what-is-fps-in-the-babble-category

Yes, FPS Babble is 30, same value for the camera and Pipe, latency 8/9

[Jeka8833]

Does the value of the cheeks and tongue parameter in the menu: https://foxyface.jeka8833.pp.ua/docs/FoxyFace/ui/calibration/Manual-Calibration change?
Is it possible that the avatar does not support cheeks or tongue? Try this avatar: https://vrchat.com/home/avatar/avtr_7b217580-e724-452f-8dca-7778738450cc

[unreal-2222]

Does the value of the cheeks and tongue parameter in the menu: https://foxyface.jeka8833.pp.ua/docs/FoxyFace/ui/calibration/Manual-Calibration change? Is it possible that the avatar does not support cheeks or tongue? Try this avatar: https://vrchat.com/home/avatar/avtr_7b217580-e724-452f-8dca-7778738450cc

The avatar definitely has full FT support. The cheek and tongue parameters in the FoxyFace menu have the following behavior: for the cheeks, the offset slider fluctuates at minimal values on its own (so you can see that it is tracking something), but overall it does not respond to the movement of my cheeks at all. With the tongue slider, the situation is slightly different: it only responds if I tilt my head upwards (tongue sticking out position), but there is no reaction to tongue rotation or offset. In VRChat, it’s the same—if I tilt my head slightly upwards, the tongue automatically sticks out and may even get stuck in the out position even if the mouth is closed.

Also, all program settings are generally at default; the only changes are the camera settings and the updated Babble.onnx. In MediaPipe preview mode, the mesh lies perfectly, and there are no problems in the Babble preview either.

[Jeka8833]

I understand your problem, the Babble neural model has difficulty seeing your face. The parameter sticking clearly indicates that your camera is positioned noticeably to the side (It is not looking directly at your face). You can fix this by moving the camera or changing the Max head up-down / left-right rotation degree parameters: https://foxyface.jeka8833.pp.ua/docs/FoxyFace/ui/babble/Babble-Settings#max-head-up-down--left-right-rotation-degree

Also, try following the basic setup instructions and performing an automatic calibration for the tongue and cheeks:

1. https://foxyface.jeka8833.pp.ua/docs/FoxyFace/ui/camera/Camera-Settings/#improving-the-quality-of-facial-tracking
2. https://foxyface.jeka8833.pp.ua/docs/FoxyFace/ui/calibration/Auto-Calibration

[unreal-2222]

My camera is mounted on top of my monitor; the monitor is on a bracket, so I can lower it to make the camera look straight at me—it’s not positioned to the side. Adjusting the calibration settings did yield some results: although I don’t fully understand the correct procedure, I slightly broke the avatar’s facial expressions but got the cheeks and tongue partially working (I couldn’t achieve a perfect result, so I decided to reset everything to default and use it as is). In any case, it’s strange that the program doesn’t work perfectly out of the box with full functionality. My friends in VRChat also tried your program and experienced the same issues with cheek and tongue tracking. The only thing I understood is that calibration takes time, and usually: when the cheeks start working, something else stops working.

[Jeka8833]

Yes, the neural network from Babble creates some problems, as I am using it for an unintended purpose. I have neither the financial resources nor the computational power to create my own neural network for tracking cheeks and the tongue. Also, I do not have permission to use the new versions of the neural network from Project Babble, which creates some inconvenience in use (It doesn't work out of the box).

[Jeka8833]

If you need help with auto-calibration let me know. Also, I am open to any PR.

[unreal-2222]

Thank you for your support, but I think I’ll keep it for myself in default mode. At least now I know the reason why the tongue and cheeks aren’t being tracked.

[Jeka8833]

There's no need to leave everything as the standard; update the neural network from Babble and recalibrate each parameter individually to avoid overwriting the calibration of already well-calibrated parameters. If you can't calibrate a parameter in automatic mode, try manual calibration.

It is also possible that increasing the resolution of the image from the camera may improve tracking.

All information can be found here: https://foxyface.jeka8833.pp.ua/docs/FirstSetup/