QVAC-18395: Continuous Batching (text, single-job)

[jesusmb1995]

The problem

When a single request is in flight, the GPU sits idle between token generation steps while the sequence waits on sampling or I/O. Nothing batches work across concurrent callers, so each request effectively serialises the whole pipeline.

What this does

This PR introduces continuous batching: a native decode loop (ContinuousBatchScheduler) that, on every step, pulls tokens from all active sequences into a single shared batch. Instead of one sequence at a time, the GPU sees a full batch drawn from however many callers are currently in flight.

Two components handle the concurrency machinery. MultiRequestBatcher manages admission and maps each incoming sequence to a slot. SequenceDriver owns the per-slot generation state: KV cache position, token budget, stop conditions. Intake runs through a lock-free moodycamel/concurrentqueue port so admission doesn't block the decode loop.

Interface

On the JS side, run() now accepts either a single Message[] (unchanged) or an array of prompts as Message[][] or BatchPrompt[] wrappers, which carry optional ids and per-item runOptions. It returns a BatchResponse that
streams BatchOutputChunk events keyed by sequence id and resolves await() to an ordered BatchResult[]. RuntimeStats gains avgConcurrentSeq to report observed decode concurrency over a run. The level of parallelism can be enabled and controlled with parallel flag when building the model.

Limitations

Text-only. Vision will be handled on a separate PR.

For now JS interface limited to single-job. This means that to truly take advantage of parallelism multiple messages needs to be bundled in same run() call. Multiple separate consecutive run calls with their own messages will not take advantage of the parallelism. This will be addressed later.

Testing

Integration tests cover basic multi-prompt batching, per-item generationParams, streaming output correlated by id, and concurrent admission. On the C++ side, unit tests cover MultiRequestBatcher slot lifecycle, the ContinuousBatchScheduler decode loop and budget enforcement, and a regression for the n_predict single-prompt guard.

API changes

• run(prompt) accepts Message[] (unchanged) or BatchPrompt[] /
  Message[][] (new batch path).
• New interfaces: BatchPrompt, BatchOutputChunk, BatchResult,
  BatchResponse.
• RuntimeStats.avgConcurrentSeq: number added.

---

See previous PR #1890

[github-actions]

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ❌ PENDING

**Requirements:**
- 1 Team Member approval ❌ (0/1)
- 1 Team Lead OR Management approval ❌ (0/1)



---
*This comment is automatically updated when reviews change.*

[gianni-cor]

Can we compute batch TPS and ppTPS from llama.cpp's aggregate context-level perf counters instead of the scheduler wall-clock timer? For batch we cannot get exact per-prompt compute time from llama.cpp, but the aggregate batch metrics can still mirror the single-prompt formulas: TPS = 1000 / perfData.t_eval_ms * perfData.n_eval and ppTPS = 1000 / perfData.t_p_eval_ms * perfData.n_p_eval. That also restores ppTPS for batch stats and keeps these fields based on llama.cpp timing rather than external elapsed time.

[jesusmb1995]

Note: this explanation was AI-generated (reviewed by me before posting).

I looked into this and I don't think the llama.cpp counters work for the batch path. Using them would make TPS and ppTPS worse than the wall-clock number we have now, so I'd push back on the suggestion.

Here's the issue. llama.cpp decides whether a decode step is prefill or generation purely from the batch size. In llama-context.cpp: n_queued_tokens == 1 goes to generation (t_eval/n_eval), > 1 goes to prompt processing (t_p_eval/n_p_eval). There's a FIXME right above that code warning the stats break when you evaluate multiple single tokens without a sync. That's continuous batching exactly. Every step here takes one token from each active sequence, so the batch is almost always > 1, and all our generation work gets logged as prompt eval.

So n_eval stays near zero. TPS = 1000 / t_eval_ms * n_eval collapses to roughly zero, and t_p_eval/n_p_eval hold prefill and generation mixed together, which makes ppTPS meaningless. The single-prompt formulas only work because llama.cpp can separate the two phases, and in batched mode it can't.

Better to measure it in the scheduler, which already knows the phase of every token. Time each llama_decode and classify the step. Pure-decode steps add their time and generated tokens to the decode totals. Pure-prefill steps add theirs to the prefill totals. For a mixed step (a newcomer prefilling while others generate), charge the whole step to decode: that pass had to run for the generators anyway, and the one prefill token rides along at near-zero cost. Then TPS = n_decode / t_decode and ppTPS = n_prefill / t_prefill, with each rate's tokens and time drawn from the same steps so they can't drift apart. If a run has no pure-prefill step, report ppTPS as N/A instead of inventing a value.

I'd keep the wall-clock number as the aggregate batch throughput, since that's the figure that actually shows what batching buys us. The per-step rates sit on top as the prefill/decode efficiency split.

Most of the data is already there: generatedTokens and promptTokens are tracked per step, so this is a per-step timer plus two accumulators next to the existing recordDecodeStep call.

[jesusmb1995]

Went with scheduler-side per-step timing instead of the llama.cpp counters: TPS now comes from decode-step time and ppTPS is restored from pure-prefill-step time.

The counters can't separate the phases under continuous batching — llama.cpp buckets a step by batch size, and every batched step is size > 1, so generation gets filed as prompt eval. From llama-context.cpp#L352-L367:

// FIXME: if multiple single tokens are evaluated without a synchronization,
// the stats will be added to the prompt evaluation stats
// this should only happen when using batch size 1 to evaluate a batch

// add the evaluation to the stats
if (n_queued_tokens == 1) {
    if (!cparams.no_perf) {
        t_eval_us += ggml_time_us() - t_compute_start_us;
    }
    n_eval++;
} else if (n_queued_tokens > 1) {
    if (!cparams.no_perf) {
        t_p_eval_us += ggml_time_us() - t_compute_start_us;
    }
    n_p_eval += n_queued_tokens;
}

So the rates are measured in the scheduler.

Fixes:

• 427c1af: specs (unit + a real-model batch run)
• ffed148: per-step timing impl

Left open in case you want to push back on the mixed-step → decode attribution.

[gianni-cor]

Can we add docs explaining the continuous-batching architecture? This PR introduces several new moving parts (ContinuousBatchScheduler, MultiRequestBatcher, per-slot TextLlmContext / SequenceDriver, queued vs active requests, per-sequence context caps, streaming ids, cache-per-slot behavior, and cancellation semantics). Architecture docs would make it much easier to understand how batch prompts flow from JS into native slots, what parallel controls, how outputs/stats are aggregated, and what is/isn't supported in batch mode (media, per-prompt cache files, tools, context shifting).