A multi-agent, confidence-aware medical diagnostic assistant — built at HackPSU.
SymptomSense analyzes text, image, and voice symptom input through a LangGraph-orchestrated pipeline of specialized agents (vision, retrieval, web search, synthesis, confidence verification) and reports a transparent, multi-dimensional confidence score instead of a single overconfident answer. Low-confidence cases are automatically flagged for human-in-the-loop (HITL) review.
This is a hackathon prototype. It is not a medical device and must not be used for real diagnosis.
- Multi-modal input — text symptom queries, medical images (e.g. chest X-rays), and voice (speech-to-text).
- Agentic pipeline — LangGraph state machine routing between vision analysis, RAG document retrieval, live web search, and answer synthesis.
- Confidence scoring — combines image-model confidence, RAG retrieval relevance, and LLM reasoning confidence into a single weighted profile.
- Human-in-the-loop safety — responses below a configurable confidence threshold are queued for expert review (
backend/data/hitl_queue/). - Hybrid knowledge retrieval — Qdrant vector search over ingested medical literature, with live Brave Search fallback for time-sensitive queries.
- Text-to-speech — Kokoro-based voice output for responses.
- Observability — structured logging (
structlog) and Prometheus metrics exposed at/metrics.
| Layer | Technology |
|---|---|
| API | FastAPI, Uvicorn, Pydantic v2 |
| Orchestration | LangGraph, LangChain |
| LLM | Google Gemini (google-generativeai), OpenRouter fallback |
| Vector store | Qdrant |
| Document parsing | Docling (primary), Unstructured (fallback) |
| Vision | PyTorch / torchvision / transformers (ViT for chest X-ray classification) |
| Embeddings / reranking | sentence-transformers (cross-encoder) |
| Speech | faster-whisper (STT), Kokoro (TTS) |
| Web search | Brave Search API, DuckDuckGo fallback |
| Frontend | Vanilla JS, HTML5/CSS3, Web Speech API |
| Ops | Docker Compose, Prometheus, structlog |
backend/ FastAPI app (app/), pyproject.toml, requirements
frontend/ Static JS/HTML/CSS client
scripts/ Ingestion, warm-start, metrics, verification scripts
tests/ pytest suite
docker/ Dockerfiles + entrypoints for backend/frontend/ollama
infra/ ngrok and Qdrant config
- Python 3.11+
- A Gemini API key (required) — see
.env.example - Optional: Brave Search API key for live web search fallback
git clone https://github.com/harshagarwalnyu/HackPSU-SymptomSense.git
cd HackPSU-SymptomSense
cp .env.example .env # fill in GEMINI_API_KEY and any optional keys
pip install uv
cd backend && uv pip install -e . --system && cd ..# Terminal 1 — backend
uvicorn backend.app.main:app --reload --host 0.0.0.0 --port 8000
# Terminal 2 — frontend
cd frontend && python -m http.server 3000Or with the provided Makefile:
make setup # install backend deps via uv
make dev # run the backend with reload
make run # docker compose up --builddocker compose up --build- Health check:
GET http://localhost:8000/health - Main inference:
POST http://localhost:8000/api/process_input - Speech-to-text:
POST http://localhost:8000/api/stt - Text-to-speech:
POST http://localhost:8000/api/tts - Metrics:
GET http://localhost:8000/metrics
Ingest the medical knowledge base before running RAG queries:
python scripts/ingest_data.pyWarm models ahead of a demo to avoid cold starts:
python scripts/warm_start.pypytest tests/MIT — see LICENSE.