Tongue-operated physical joysticks exist, but I'm talking about detecting your tongue's position in the camera, and mapping that to inputs.
For an initial pass, sending arrow keys could allow playing many games! As well as navigating webpages, moving the text cursor, and many other tasks.
Tongue tracking of course isn't supported by facemesh.
It's also not compatible with open mouth detection, because when your tongue is out and downwards it causes it to think your mouth is open . (That's actually an alternate way you can click currently, by accident; good if you like feeling like a lizard. More exploration is needed, but upon initial testing, it's fun to feel like a lizard.)
It would need a separate computer vision system.
This could be done using a neural network trained on images cropped to the mouth region.
Imagine training it by holding your tongue in a particular pose, then moving your head all around, and as you move your head, the system captures images of your mouth any time it finds a significantly different angle, showing you the coverage of angles by placing captures into a visual array, similar to how a panorama mode on a camera works (but not stitching the images together).
It could even be trained at runtime (similar to Teachable Machine), although we would need to update the privacy policy if mouth images are stored. So ideally a model should be distributed that works with varying skin tones under varying lighting conditions etc.
(But the training part does sound fun to me. You could even include flipping animations (if it's tile based but it probably shouldn't be) and sound effects...)
Tongue-operated physical joysticks exist, but I'm talking about detecting your tongue's position in the camera, and mapping that to inputs.
For an initial pass, sending arrow keys could allow playing many games! As well as navigating webpages, moving the text cursor, and many other tasks.
Tongue tracking of course isn't supported by facemesh.
It's also not compatible with open mouth detection, because when your tongue is out and downwards it causes it to think your mouth is open . (That's actually an alternate way you can click currently, by accident; good if you like feeling like a lizard. More exploration is needed, but upon initial testing, it's fun to feel like a lizard.)
It would need a separate computer vision system.
This could be done using a neural network trained on images cropped to the mouth region.
Imagine training it by holding your tongue in a particular pose, then moving your head all around, and as you move your head, the system captures images of your mouth any time it finds a significantly different angle, showing you the coverage of angles by placing captures into a visual array, similar to how a panorama mode on a camera works (but not stitching the images together).
It could even be trained at runtime (similar to Teachable Machine), although we would need to update the privacy policy if mouth images are stored. So ideally a model should be distributed that works with varying skin tones under varying lighting conditions etc.
(But the training part does sound fun to me. You could even include flipping animations (if it's tile based but it probably shouldn't be) and sound effects...)