GestureCue: Hand-Controlled Visual Presentations
A macOS accessibility-focused presentation and visual education app controlled with webcam hand gestures, with AI-assisted lesson creation.
GestureCue turns a Mac webcam feed into a hands-free presentation interface. Hand gestures control visual cards while the app composites those cards into the camera output, so a presenter can keep both the signer and the lesson content visible in a video call.
The project is designed with Deaf and hard-of-hearing audiences in mind. Its AI Visual Education Studio can turn source material into illustrated lesson cards, while the presentation workflow can also import existing PDF and PowerPoint material.
Project artifacts
- Contributor: shaedis
- Source code on GitHub
- Project demonstration on YouTube
What it does
- Uses webcam hand poses to show, hide, navigate, and expand presentation cards.
- Uses stillness-gated holds and a command latch to reduce accidental gesture activation while signing.
- Creates illustrated lesson decks from pasted text or imported documents with the Meta API.
- Offers a faithful presentation mode for imported material, preserving source text and original images.
- Composites cards with the live camera feed for use in Zoom and Teams, with an OBS capture fallback.
- Supports live annotation, presentation libraries, and user-initiated MP4 recording.
Architecture
The macOS app is written in Swift. Its shared capture pipeline feeds hand-pose detection, gesture classification, card presentation, video compositing, and recording. The README describes AVFoundation capture, Vision hand-pose detection, geometric gesture classification, and a state machine that gates actions on stable holds. AI-generated lesson content uses the Meta API; diagrams and other local presentation elements are composed on-device.
Privacy and limitations
According to the project README, webcam video is processed on-device and is not sent to Meta. Only source material and illustration prompts for approved lessons are sent after the user approves them. The app does not use the microphone. MP4s are created only when the user explicitly records.
The current implementation supports one detected hand at a time, works best with good lighting and a plain background, and uses geometric gesture thresholds rather than a trained classifier. The README also notes that unusual hand shapes or camera angles may be classified as unknown, and that Zoom/Teams virtual-camera behavior should be verified on the target setup.
Implementation evidence
The public repository includes the Swift app, architecture and setup documentation, and an extensive XCTest suite. The README reports 365 tests passing. The linked demo provides a visual walkthrough of the project.