Skip to content

README.md

Latest commit

 

History

History
158 lines (114 loc) · 6.36 KB

README.md

File metadata and controls

158 lines (114 loc) · 6.36 KB
Raw

Silent Talk Logo

Silent Talk

ASL Fingerspelling Recognition Prototype

Watch Demo Video

Samsung Award
Contributors License LinkedIn

Table of Contents
  1. About The Project
  2. System Architecture
  3. Getting Started
  4. Status & Limitations
  5. License
  6. Contact

About The Project

Silent Talk is a webcam prototype that classifies ASL fingerspelled letters and displays the accumulated text. It recognizes hand poses from individual frames; it does not translate continuous sign language or provide two-way interpretation.

This project was selected as a Top 30 Semifinalist (out of 300+ teams) in the Samsung Solve for Tomorrow 2024 competition.

  • Reported training data: A custom dataset of ~50,000 images.
  • Reported competition result: >90% accuracy on validation samples during competition trials. Training scripts, the validation split, and evaluation results are not included here, so this figure cannot be reproduced from the repository alone.
  • Execution: Hand tracking and letter classification run locally without requiring a GPU. Text spacing uses an external OpenAI API call.

(back to top)

Built With

  • Python
  • React
  • OpenCV
  • MediaPipe
  • Scikit-Learn

(back to top)

System Architecture

The prototype has a Flask backend and a React frontend.

1. Vision Engine (Backend)

Located in backend/, the recognition flow is:

  • Hand Tracking: MediaPipe extracts skeletal landmarks (21 points per hand) in real-time.
  • Letter classification: A saved classifier predicts one of 26 letter labels from hand landmarks. A prediction must remain stable for 1.5 seconds before it is appended.
  • Text spacing: The backend sends accumulated text to gpt-4o-mini with instructions to insert spaces. There is no speech-synthesis implementation in this snapshot.

2. User Interface (Frontend)

Located in /frontend, the client is built with React:

  • Display: Shows the backend camera feed and polls recognized text once per second.
  • Controls: Includes camera-display and text-reset controls, plus a light/dark theme.

(back to top)

Getting Started

The original setup targets Python 3.9+ and Node.js. The backend needs access to the host computer’s camera, the bundled backend/model.p, and an OPENAI_API_KEY environment variable. Starting it without an API key fails when the OpenAI client is created.

Prerequisites

  • Python 3.9+
  • Node.js & npm

Installation

  1. Clone the repository
git clone https://github.com/alexandr-tk/silent-talk.git
  1. Setup Backend (Python)
cd silent-talk/backend
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows
pip install -r requirements.txt
export OPENAI_API_KEY="<your-api-key>"  # backend environment only
python app.py
  1. Setup Frontend (React), in a second terminal from the repository root
cd frontend
npm install
npm run dev

(back to top)

Status & Limitations

Status: Competition proof of concept

This codebase represents the competition snapshot of Silent Talk submitted for the Samsung Solve for Tomorrow semifinals.

  • Code Quality: Experimental / Prototype Grade.
  • Maintenance: This competition snapshot is not currently maintained.
  • Integration limits: The settings client calls /set_stable_time, but the Flask backend does not implement that route. Some service helpers use relative URLs while Vite has no API proxy. The camera and main text display use localhost:5000 directly.

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Contact

Alex Tkachyov - Lead Developer - LinkedIn

Project Link: Silent Talk on GitHub

(back to top)