QVAC-18395: Continuous Batching (text, single-job)#2327

Merged

jesusmb1995 merged 28 commits into

tetherto:mainfrom

jesusmb1995:continuousBatching6

Jun 15, 2026

Commits on Jun 15, 2026

Continuous Batching (text, single-job) squashed
jesusmb1995
committed
Relax prefill-vs-decode timing assertion in continuous batching test
jesusmb1995
committed
Replace parameter comments with constexpr declarations
jesusmb1995
committed
Constrain BatchPrompt.prompt to text messages and function definitions
jesusmb1995
committed
test: reproduce stale cancel state across batch/single modes
jesusmb1995
committed
fix: route cancel to the active engine to avoid stale cancel state
jesusmb1995
committed
fix: accept mixed Message[]/BatchPrompt arrays in run()
jesusmb1995
committed
test: reproduce KV cache leak on batch admission failure after loadCache
jesusmb1995
committed
test: reproduce CacheManager stale residency after batch clears seq 0
jesusmb1995
committed
fix(comment-3401534906): invalidate CacheManager session state when batching runs
jesusmb1995
committed
test: reproduce DecodeError completing batch on success path
jesusmb1995
committed
fix(comment-3401387685): propagate decode errors via failGroupLocked instead of success path
jesusmb1995
committed
fix(comment-3401535953): prevent KV cache leak on batch admission failure after loadCache
jesusmb1995
committed
test: reproduce reload() cancel state bug (RED)
jesusmb1995
committed
fix: unconditionally stop contexts in reload() (GREEN)
jesusmb1995
committed
Fixup: Formatting after recently introduced fixed for review comments
jesusmb1995
committed
fix: remove incorrect namespace qualifications in test
jesusmb1995
committed
doc: clarify global min chunk is a deliberate simplicity tradeoff
jesusmb1995
committed
doc: clarify workerLoop catch(...) is last-resort fail-all by design
jesusmb1995
committed
fix: make reload-cancel test compile and run against the real API
jesusmb1995
committed
test: reproduce batch-mode context slide never firing (RED)
jesusmb1995
committed
fix: wire per-sequence context sliding into batch mode (GREEN)
jesusmb1995
committed
test: reproduce cancel/clear mutating context during unlocked decode (RED)
jesusmb1995
committed
fix: funnel cancel(seqId)/clear() through the worker thread (GREEN)
jesusmb1995
committed
fixup: Formatting C++ files
jesusmb1995
committed
doc(comment-3412030323): fix batch README example to match runtime API
jesusmb1995
committed
doc(comment-3412030323): clarify batch response ids

authored and
jesusmb1995
committed
fixup: Bump version
jesusmb1995
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

QVAC-18395: Continuous Batching (text, single-job)#2327

QVAC-18395: Continuous Batching (text, single-job)#2327
jesusmb1995 merged 28 commits into
tetherto:mainfrom
jesusmb1995:continuousBatching6

Commits on Jun 15, 2026

Continuous Batching (text, single-job) squashed

Relax prefill-vs-decode timing assertion in continuous batching test

Replace parameter comments with constexpr declarations

Constrain BatchPrompt.prompt to text messages and function definitions

test: reproduce stale cancel state across batch/single modes

fix: route cancel to the active engine to avoid stale cancel state

fix: accept mixed Message[]/BatchPrompt arrays in run()

test: reproduce KV cache leak on batch admission failure after loadCache

test: reproduce CacheManager stale residency after batch clears seq 0

fix(comment-3401534906): invalidate CacheManager session state when batching runs

test: reproduce DecodeError completing batch on success path

fix(comment-3401387685): propagate decode errors via failGroupLocked instead of success path

fix(comment-3401535953): prevent KV cache leak on batch admission failure after loadCache

test: reproduce reload() cancel state bug (RED)

fix: unconditionally stop contexts in reload() (GREEN)

Fixup: Formatting after recently introduced fixed for review comments

fix: remove incorrect namespace qualifications in test

doc: clarify global min chunk is a deliberate simplicity tradeoff

doc: clarify workerLoop catch(...) is last-resort fail-all by design

fix: make reload-cancel test compile and run against the real API

test: reproduce batch-mode context slide never firing (RED)

fix: wire per-sequence context sliding into batch mode (GREEN)

test: reproduce cancel/clear mutating context during unlocked decode (RED)

fix: funnel cancel(seqId)/clear() through the worker thread (GREEN)

fixup: Formatting C++ files

doc(comment-3412030323): fix batch README example to match runtime API

doc(comment-3412030323): clarify batch response ids

fixup: Bump version