Mac Finger Game Control

Personal prototype built in July 2026. It works on my machine and I use it for fun — it is not a polished or maintained product.

Native macOS camera hand-control prototype for Roblox-style games. It uses Apple Vision hand pose tracking and synthesizes real keyboard key down/up events (CGEvents), which macOS only allows after the app is granted the Accessibility permission.

This project is inspired by minwoo19930301/mac-hand-control, but it targets game movement instead of Spaces/fullscreen shortcuts.

Gestures

• Fold index finger: holds A.
• Fold middle finger: holds W.
• Fold ring finger: holds D.
• Fold pinky finger: holds S.
• Fold two or three fingers: holds the matching movement keys together (folding all four becomes the fist gesture instead).
• Thumb-index pinch: taps Space for jump.
• Thumb-middle pinch: taps I for Roblox-style zoom in.
• Thumb-ring pinch: taps O for Roblox-style zoom out.
• Fist: taps E (and releases any held movement keys).
• All fingers spread out: releases movement keys and stops.

Demo

How it works

Everything lives in one Swift file (Sources/MacFingerGameControl/main.swift, ~1,450 lines) with no third-party dependencies — just Apple frameworks: AVFoundation, Vision, CoreGraphics, ApplicationServices, CoreImage, and AppKit.

Pipeline

1. Capture — an AVCaptureSession (.medium preset) reads the front builtInWideAngleCamera (falling back to the system default camera if that isn't available) through an AVCaptureVideoDataOutput with alwaysDiscardsLateVideoFrames, delivering frames on a serial dispatch queue. Frames are throttled to at most ~30 fps.
2. Hand pose — each frame goes through a VNImageRequestHandler running a VNDetectHumanHandPoseRequest with maximumHandCount = 2. Joints come from recognizedPoints(.all) and are kept only when confidence is at least 0.30. Only the first detected hand drives the controls.
3. Gesture classification — a small state machine (GameGestureController) turns joint positions into intent:
   • Finger bend: for each finger, extension = distance(wrist, tip) − distance(wrist, PIP), normalized by a palm scale (wrist-to-index-MCP distance). A finger counts as extended above 0.14× palm scale (0.12× for the pinky) and folded below 0.10× (0.08× for the pinky); in between it is neutral, which acts as a dead band so a half-bent finger doesn't flicker between key press and release.
   • Pinch: the thumb tip's distance to the index/middle/ring tips is normalized by palm width (index-MCP to little-MCP); the closest tip within 0.46× palm width counts as a pinch. Pinch taps are edge-triggered with a 0.45 s cooldown, and a pinched finger's movement key is suppressed so pinching for Space doesn't also hold A.
   • Fist / stop: all four fingers folded taps E (edge-triggered, 0.55 s cooldown); all four extended is a stop pose that releases every held key.
4. Keyboard output — a KeyboardDriver creates keyboard CGEvents from a CGEventSource(stateID: .hidSystemState) and posts them to .cghidEventTap. Held movement keys are updated as a set diff (key-down for newly folded fingers, key-up for released ones), and taps are key-down + 35 ms + key-up. Nothing is posted unless AXIsProcessTrusted() returns true; the app prompts via AXIsProcessTrustedWithOptions and links to the Accessibility pane.
5. Debug UI — an AppKit window (floating level, joins all Spaces) shows the mirrored camera feed (rendered to CGImage via CIContext), a hand-skeleton overlay drawn with NSBezierPath joint chains, green badges on fingertips classified as folded (which normally means their movement key is held — except a pinch-suppressed finger), pinch/fist/STOP readouts, and a status panel. The same Pause / Reset / Release actions are also available from an NSStatusItem menu-bar item, so the window can be closed while playing.

Build

Requires macOS 14+ and the Xcode Command Line Tools (swiftc).

./build.sh

The app bundle is created at:

build/Mac Finger Game Control.app

Use

1. Open the app from build/Mac Finger Game Control.app.
2. Approve Camera permission.
3. Click Enable AX and allow the app in System Settings > Privacy & Security > Accessibility.
4. Put Roblox or another game in front.
5. Fold individual fingers as button controls.

The debug window can stay floating while you play, or you can close it and keep the menu bar item running.

Package

./package.sh

This creates:

dist/Mac-Finger-Game-Control.dmg

Limitations

• Only the first detected hand sends keys; a second hand is drawn in the overlay but ignored for control.
• The key mapping (W/A/S/D, Space, E, I, O) and all gesture thresholds are hardcoded — there is no remapping UI and no per-user calibration.
• Pinch and fist taps are rate-limited by fixed cooldowns (0.45 s and 0.55 s), so rapid repeated taps aren't possible.
• Keyboard only: folded fingers map to on/off key holds. There is no mouse/camera-look control and no analog input.
• Tracking needs the wrist, the index MCP (the palm-scale reference), and all four finger tip/PIP joints visible above the confidence threshold; if any drop out, the app falls back to "waiting" and releases all keys.
• Frames are processed at most ~30 fps, so there is noticeable latency between a gesture and the key event.
• macOS 14+ only. build.sh compiles for the host architecture only (not a universal binary), and games that read input below the HID event tap may ignore synthetic CGEvents.

Notes

• If a gesture gets confused, use Reset in the app or menu bar.
• If a key seems stuck, use Release or spread all fingers out.
• Some games may require you to click back into the game after opening the debug window.
• This is ad-hoc signed for local use, so macOS may show an extra security prompt the first time you run it.
• Everything runs on-device; the app has no networking code and never records or uploads camera frames.