Implement in-memory LRU cache for audio synthesis requests

[hdduytran]

feat: in-memory LRU cache for repeated synthesis requests

Problem

Some clients send the same request repeatedly — same clone profile, same text, same parameters. Each request runs full inference (~10s on CPU, ~0.5s on GPU), wasting compute on identical work.

Solution

An in-memory LRU cache stores the final audio bytes (WAV/PCM) keyed on all output-affecting parameters. Identical requests return instantly from cache with zero inference cost.

How it works

On each non-streaming /v1/audio/speech request, the router hashes (voice, text, speed, num_step, guidance_scale, denoise, t_shift, position_temperature, class_temperature, duration, response_format) into a SHA-256 cache key. The voice field is the raw client string (e.g. "clone:voice11") — a stable semantic identifier that doesn't depend on filesystem paths.

Cache hits return stored bytes immediately with X-Cache: HIT. Misses run inference normally, store the result, and respond with X-Cache: MISS.

Memory safety:

• LRU eviction when total bytes exceed cache_max_mb (default 512MB)
• TTL expiry via background sweep task (interval = cache_ttl_s / 2, min 10s)
• Lazy TTL check on get() as well
• Entries larger than the entire budget are silently skipped (prevents thrashing)
• Cache stores bytes, not GPU tensors — no VRAM impact

Not cached: streaming requests (can't cache a chunked stream), one-shot /v1/audio/speech/clone (temp file paths change every request).

Changes

• New omnivoice_server/services/cache.py — AudioCache with LRU OrderedDict, TTL sweep task, _build_cache_key()
• config.py — cache_enabled, cache_max_mb, cache_ttl_s fields
• app.py — wire AudioCache into lifespan (start/stop)
• routers/speech.py — cache lookup before inference, cache store after, X-Cache header, DELETE /v1/audio/cache endpoint
• cli.py — --cache-enabled, --no-cache, --cache-max-mb, --cache-ttl-s flags
• health.py — expose cache config in /health, cache stats in /metrics
• README, docs/architecture/overview.md, docs/design/dataflow.md — updated diagrams, request lifecycle, endpoint docs

Configuration

Flag	Env var	Default	Description
--cache-enabled / --no-cache	OMNIVOICE_CACHE_ENABLED	true	Enable/disable audio cache
--cache-max-mb	OMNIVOICE_CACHE_MAX_MB	512	Max cache memory (LRU eviction)
--cache-ttl-s	OMNIVOICE_CACHE_TTL_S	3600	Entry TTL in seconds (0 = no expiry)

API

# Check cache stats
curl http://localhost:8880/metrics
# → { "cache_hits": 42, "cache_hit_rate": 0.808, ... }

# Clear cache
curl -X DELETE http://localhost:8880/v1/audio/cache

Testing

All 40 existing tests pass. Cache is created during app lifespan but tests mock synthesize() directly so cache integration is transparent.

[maemreyo]

Hi @hdduytran, great implementation on the LRU cache! 🎉

I'm curious about the 512MB default for cache_max_mb. With 24kHz WAV files (~48KB/s), this holds roughly ~1000 x 10s audio clips. For production workloads with many voice profiles, we might hit the limit quickly.

Thoughts on bumping default to 2048MB (2GB)? Modern servers typically have 16GB+ RAM, and 2GB would cache ~4000 entries without being too aggressive. Would love your take on this!

[maemreyo]

Hi @hdduytran, great implementation on the LRU cache!

I'm curious about the 512MB default for cache_max_mb. With 24kHz WAV files (~48KB/s), this holds roughly ~1000 x 10s audio clips. For production workloads with many voice profiles, we might hit the limit quickly.

Thoughts on bumping default to 2048MB (2GB)? Modern servers typically have 16GB+ RAM, and 2GB would cache ~4000 entries without being too aggressive. Would love your take on this!

[maemreyo]

The cache logic looks solid! To make future maintenance easier, would you be open to adding a tests/test_cache.py with a few unit tests? I'm thinking:

• LRU eviction when budget exceeded
• TTL expiry behavior
• Hit/miss counters
• Oversized entry skip

Happy to help write these if you're busy - just want to ensure the logic stays robust as we iterate.

[maemreyo]

With ttl_s=3600, the sweep runs every 30 min (max(3600/2, 10)). Stale entries could linger up to 30 min after expiry.

What do you think about making the divisor configurable or defaulting to 4 instead of 2? That would sweep every 15 min - more memory-efficient without much overhead. Or is the 30-min window intentional for performance reasons?

[maemreyo]

The cache key uses voice="clone:profile_id" which is clean! One edge case: if a user updates the reference audio for a profile (same ID, different voice), the cache would still return old audio until TTL/LRU eviction.

Options I'm considering:

1. Document this in README (cache clear recommended after profile updates)
2. Add ProfileService hook to auto-clear affected cache entries
3. Include profile file hash in cache key (adds complexity)

What's your take? For v1, option 1 (documentation) seems reasonable - we could add a note in the "Audio Cache" section.

[maemreyo]

Could we add a cache_skipped_oversized counter for entries that exceeded cache_max_mb? This would help operators tune the cache size when they see frequent skips in metrics.

This came to mind while reviewing the put() method - it's great that we skip huge entries, but having visibility into how often it happens would be valuable for tuning.

[maemreyo]

Hi @hdduytran Im adding some feedback when walking through your excellent work. Please feel free to reach out, we can discuss more bout those.
Once again, thanks for your dedicated work on this!!