Implement dynamic batching for inference and update documentation

[hdduytran]

feat: dynamic batching for inference

Problem

Under concurrent load, each request runs as an independent model.generate() call. On GPU, this serializes requests through the thread pool — the GPU processes one at a time while others wait on the semaphore. Throughput scales linearly with request count.

Solution

Leverage OmniVoice's native batch API (model.generate(text=[...], ref_audio=[...], ...)) to group concurrent requests into a single GPU call. A batch of 8 requests takes roughly the same wall time as 2-3 individual calls.

How it works

A background BatchingService accumulates incoming requests into parameter-compatible groups. A batch dispatches when either batch_max_size (default 8) is reached or batch_timeout_ms (default 50ms) elapses. Each request gets an asyncio.Future and awaits its individual result.

Requests with different generation parameters (num_step, guidance_scale, etc.) are grouped separately and dispatched independently. Per-item parameters (text, voice, speed, duration) vary freely within a batch.

A batch_semaphore caps concurrent batch dispatches to max_concurrent, preventing unbounded executor queue growth. If the batched call fails (e.g. upstream API change), it falls back to sequential single-item generation.

Changes

• New omnivoice_server/services/batching.py — BatchingService with dispatch loop, parameter grouping, sequential fallback
• inference.py — InferenceService delegates to BatchingService when enabled; legacy semaphore path preserved for --no-batch
• config.py — batch_enabled, batch_max_size, batch_timeout_ms fields; clarified max_concurrent semantics
• app.py — wire BatchingService into lifespan (start/stop)
• cli.py — --batch-enabled, --no-batch, --batch-max-size, --batch-timeout-ms flags
• metrics.py — batches_dispatched, avg_batch_size counters
• health.py — expose batch config in /health
• tests/conftest.py — batch_enabled=False in test fixture (tests mock synthesize directly)
• README, docs/architecture/overview.md, docs/design/dataflow.md — updated diagrams, concurrency model, config table

Configuration

Flag	Env var	Default	Description
--batch-enabled / --no-batch	OMNIVOICE_BATCH_ENABLED	true	Enable/disable dynamic batching
--batch-max-size	OMNIVOICE_BATCH_MAX_SIZE	8	Max requests per batch
--batch-timeout-ms	OMNIVOICE_BATCH_TIMEOUT_MS	50	Accumulation window before dispatch

Testing

All 40 existing tests pass. Batching is disabled in test fixtures since tests mock synthesize() directly.

[maemreyo]

Hi @hdduytran, the batching implementation looks excellent! I'm curious about the
50ms default timeout - it feels quite tight for production workloads.

My thinking: With 50ms, we might miss batching opportunities due to network jitter
or request arrival patterns. For example, if requests arrive at 60ms intervals, we'd
dispatch most batches as size-1 instead of accumulating.

Options to consider:

1. Bump default to 100ms for better batching efficiency
2. Or implement adaptive timeout that scales with load (lower timeout under high
   load, higher under low load)

What's your take on this? Have you tested the batching efficiency with different
timeout values?

[maemreyo]

I noticed a potential issue in the tensor result processing logic:

t_list = [tensor] if tensor.dim() <= 2 else [tensor]

Question: Both branches return [tensor] - was the else branch intended to do something different? Perhaps:

t_list = [tensor] if tensor.dim() <= 2 else tensor.unbind(0)

Or is this the intended behavior for how OmniVoice returns batched results? Just want to verify this is correct before merging.

[maemreyo]

The parameter-based grouping is smart! I'm wondering if it's too conservative for some use cases.

Example: num_step=32 and num_step=28 can't batch together, even though GPU would handle them similarly. The difference in quality is small, but the batching efficiency gain could be significant.

Idea for v2: "Fuzzy grouping" with configurable tolerance - e.g., num_step within ±4 steps could pad to the same value. This trades a tiny bit of quality for much better throughput.

For now, the current approach is safer - just wanted to plant this seed for future optimization!

[maemreyo]

The delegation pattern is clean! One optimization idea:

Current behavior: Even with just 1 request and empty queue, we still wait batch_timeout_ms before dispatching.

Potential optimization: "Fast path" that checks if queue is empty and immediately dispatches single requests without waiting:

if self._batching_svc is not None:
    if self._batching_svc.is_empty() and self._batching_svc.pending_count() == 0:
        return await self._synthesize_single(req)  # Skip batching overhead
    return await self._batching_svc.submit(req)

This would eliminate the 50ms latency penalty for low-traffic scenarios. Worth considering?

[maemreyo]

The fallback mechanism is great defensive coding! One refinement suggestion:

Current: Catches all TypeError and falls back to sequential.
Risk: Could mask actual bugs in kwargs construction.

Suggestion: Only fallback on batch-specific errors:

except TypeError as exc:
    if "batch" in str(exc).lower() or "list" in str(exc).lower():
        logger.warning(f"Batched generate failed, falling back: {exc}")
        return self._fallback_sequential(batch, param_key)
    raise  # Re-raise if unrelated TypeError

This prevents masking unrelated bugs while keeping the safety net for API changes.

[maemreyo]

The batch_max_size=8 default is reasonable for consumer GPUs. For the H100/A100 deployments mentioned in Issue #4, this might be conservative.

Data point: A100/H100 can often handle batches of 16-32 concurrently with the same latency as batch of 8.

Suggestion: Either:

1. Document that enterprise GPU users should tune this to 16-32
2. Or add device-specific defaults (complex but user-friendly)

For now, a README note would be helpful: "If using A100/H100, consider increasing --batch-max-size to 16-32 for optimal throughput."

[maemreyo]

The documentation is clear! One small question:
"a batch of 8 requests takes roughly the same time as 2-3 individual calls"

Is this based on:

1. Actual benchmarks you've run?
2. Upstream OmniVoice documentation?
3. Estimate based on typical GPU behavior?

If it's an estimate, maybe add "(estimated)" to set expectations. If it's from actual testing, that's great data to have!

[maemreyo]

Hi @hdduytran Im adding some feedback when walking through your excellent work. Please feel free to reach out, we can discuss more bout those.
Once again, thanks for your dedicated work on this!!
Same as the #5