Silent Talk

ASL Fingerspelling Recognition Prototype

Table of Contents

1. About The Project
   • Built With
2. System Architecture
3. Getting Started
   • Prerequisites
   • Installation
4. Status & Limitations
5. License
6. Contact

About The Project

Silent Talk is a webcam prototype that classifies ASL fingerspelled letters and displays the accumulated text. It recognizes hand poses from individual frames; it does not translate continuous sign language or provide two-way interpretation.

This project was selected as a Top 30 Semifinalist (out of 300+ teams) in the Samsung Solve for Tomorrow 2024 competition.

• Reported training data: A custom dataset of ~50,000 images.
• Reported competition result: >90% accuracy on validation samples during competition trials. Training scripts, the validation split, and evaluation results are not included here, so this figure cannot be reproduced from the repository alone.
• Execution: Hand tracking and letter classification run locally without requiring a GPU. Text spacing uses an external OpenAI API call.

(back to top)

Built With

• 
• 
• 
• 
• 

(back to top)

System Architecture

The prototype has a Flask backend and a React frontend.

1. Vision Engine (Backend)

Located in backend/, the recognition flow is:

• Hand Tracking: MediaPipe extracts skeletal landmarks (21 points per hand) in real-time.
• Letter classification: A saved classifier predicts one of 26 letter labels from hand landmarks. A prediction must remain stable for 1.5 seconds before it is appended.
• Text spacing: The backend sends accumulated text to gpt-4o-mini with instructions to insert spaces. There is no speech-synthesis implementation in this snapshot.

2. User Interface (Frontend)

Located in /frontend, the client is built with React:

• Display: Shows the backend camera feed and polls recognized text once per second.
• Controls: Includes camera-display and text-reset controls, plus a light/dark theme.

(back to top)

Getting Started

The original setup targets Python 3.9+ and Node.js. The backend needs access to the host computer’s camera, the bundled backend/model.p, and an OPENAI_API_KEY environment variable. Starting it without an API key fails when the OpenAI client is created.

Prerequisites

• Python 3.9+
• Node.js & npm

Installation

1. Clone the repository

git clone https://github.com/alexandr-tk/silent-talk.git

2. Setup Backend (Python)

cd silent-talk/backend
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows
pip install -r requirements.txt
export OPENAI_API_KEY="<your-api-key>"  # backend environment only
python app.py

3. Setup Frontend (React), in a second terminal from the repository root

cd frontend
npm install
npm run dev

(back to top)

Status & Limitations

Status: Competition proof of concept

This codebase represents the competition snapshot of Silent Talk submitted for the Samsung Solve for Tomorrow semifinals.

• Code Quality: Experimental / Prototype Grade.
• Maintenance: This competition snapshot is not currently maintained.
• Integration limits: The settings client calls /set_stable_time, but the Flask backend does not implement that route. Some service helpers use relative URLs while Vite has no API proxy. The camera and main text display use localhost:5000 directly.

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Contact

Alex Tkachyov - Lead Developer - LinkedIn

Project Link: Silent Talk on GitHub

(back to top)