Skip to content

Execution Layer

Devin edited this page Apr 27, 2026 · 8 revisions

Execution Layer

Optional. Skip the whole layer and drive VulkanSync + VulkanCommand by hand if you prefer — every uninitialised module costs zero (rule 19). When you opt in, you get a frame loop, a CPU job graph, batched vkQueueSubmit, and a debug timeline.

Header source of truth: layers/execution/VCKExecution.h.

The scheduler never calls vkAcquireNextImageKHR or vkQueuePresentKHR (rule 7). The user owns the frame.


FramePolicy

Three policies. Pick one at scheduler init.

Policy Behaviour When to use
Lockstep CPU waits for GPU every frame. Deterministic, slow. Profiling / capture. Reproducible bug repros.
Pipelined CPU N+1, GPU N. Standard double-buffering. Default. Almost always this.
AsyncMax CPU may run up to cfg.asyncMaxLag frames ahead. Needs BackpressureGovernor. Latency-tolerant pipelines (offline rendering, video encode).
FrameScheduler::Config cfg;
cfg.policy      = FramePolicy::Pipelined;   // default
cfg.asyncMaxLag = 2;                        // only matters for AsyncMax

Decision guide: default to Pipelined until measurement says otherwise. Lockstep is for the debugger. AsyncMax is for when the GPU is the bottleneck and you can buffer more frames of input without it being noticeable.


TimelineSemaphore

Wrapper over VK_KHR_timeline_semaphore. A 64-bit monotonic counter signalled by the GPU and waitable from CPU or another submit. Replaces fences + binary semaphores for cross-queue ordering.

if (device.HasTimelineSemaphores()) {
    TimelineSemaphore ts;
    ts.Initialize(device, /*initialValue=*/0);

    uint64_t now = ts.LastSignaledValue();
    bool ok      = ts.Wait(/*value=*/42);          // CPU waits up to UINT64_MAX ns
    ts.Signal(43);                                 // host-side signal (rare)

    ts.Shutdown();
}

Always check device.HasTimelineSemaphores() first. Initialize returns false if the feature isn't enabled — fall back to VulkanSync's binary fences.


DependencyToken

A (TimelineSemaphore*, uint64_t) pair. Produced by whoever submits GPU work; consumed by whoever needs to wait. Invalid tokens are a no-op on wait, so hot paths don't have to check whether a producer ran.

DependencyToken tok = scheduler.SlotToken(slot);   // resolves when slot's gfx retires

if (tok.WaitHost(/*timeoutNs=*/16'000'000)) {
    // GPU finished slot's frame
}

Used by JobGraph to express "compute job depends on graphics frame N".


QueueSet

Bundle of the four queues VCK knows about: graphics, present, compute, transfer. FrameScheduler owns one and exposes it via scheduler.Queues(). Compute / transfer alias graphics on hardware without dedicated families (rule 1: no surprises — VCK logs which alias happened via VCKLog::Notice("QueueSet", ...)).


GpuSubmissionBatcher

Collects N command buffers into one vkQueueSubmit per queue. Reduces driver overhead and lets the scheduler wire frame semaphores in one place.

GpuSubmissionBatcher::SubmitInfo info;
info.waitSem   = sync.GetImageAvailableSemaphore(slot);
info.waitStage = VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT;
info.signalSem = sync.GetRenderFinishedSemaphore(slot);

f.Submissions().QueueGraphics(f.PrimaryCmd(), info);
f.Submissions().QueueCompute (computeCmd);                 // no extra sync
f.Submissions().QueueTransfer(transferCmd);

f.Submissions().FlushAll(sync.GetInFlightFence(slot));     // fires three vkQueueSubmits

FlushAll collapses everything queued since the last flush into one submit per queue. SubmitInfo carries one wait semaphore, one wait stage, and one signal semaphore — the common render-then-present case. The convenience overload f.QueueGraphics(info) is the same as f.Submissions().QueueGraphics(f.PrimaryCmd(), info). The no-arg f.QueueGraphics() wires ImageAvailableRenderFinished for you.


BackpressureGovernor

Tracks how far CPU is ahead of GPU and stalls when policy says so. Lockstep stalls every frame; Pipelined allows one frame ahead; AsyncMax allows up to cfg.asyncMaxLag. Read scheduler.Governor().Policy() / LastRetiredFrame() for diagnostics.

You don't call this directly — BeginFrame / EndFrame consult it.


JobGraph

Per-frame CPU job graph. Submit work that runs on the worker pool sized by cfg.jobWorkers (defaults to hardware_concurrency). Drained at EndFrame.

JobGraph& jg = f.Jobs();

JobGraph::JobId a = jg.Add("UpdateAnimations", []{ /* ... */ });
JobGraph::JobId b = jg.Add("CullScene",        []{ /* ... */ }, { a });
JobGraph::JobId c = jg.Add("RecordSecondary",  []{ /* ... */ }, { b });

scheduler.DispatchJobs();   // calls jg.Execute(); blocks until everything ran
// ... record GPU work in main thread, consuming b/c outputs ...

scheduler.DispatchJobs() runs the per-frame graph and returns when the graph is drained. Reset() is called for you at the next BeginFrame.


DebugTimeline

Append-only span recorder keyed by absolute frame number. Opt-in: call Initialize(true) to start recording; every method is a cheap no-op when disabled (rule 19).

DebugTimeline& dt = scheduler.Timeline();
dt.Initialize(/*enabled=*/cfg.debug);

uint64_t fn = scheduler.AbsoluteFrame();

dt.BeginCpuSpan("RecordCmds", fn);
//   ... record vkCmd* into f.PrimaryCmd() ...
dt.EndCpuSpan("RecordCmds", fn);

// Diagnose forced waits without instrumenting the source of the wait:
dt.NoteStall("swapchain recreate", fn, /*durationUs=*/2500);

// Dump to log (clears spans), or export Chrome-tracing JSON.
dt.Dump();
dt.DumpChromeTracing("/tmp/vck.trace.json");   // load in chrome://tracing or perfetto

HandleLiveResize scheduler overloads write a span to the same timeline so the recreation event chain is observable (rule 12).


Frame

Lightweight handle returned by scheduler.BeginFrame(). Lifetime: BeginFrameEndFrame. Don't keep references past EndFrame.

Frame& f = scheduler.BeginFrame();

uint32_t        slot = f.Slot();          // 0 .. MAX_FRAMES_IN_FLIGHT-1
uint64_t        abs  = f.Absolute();      // monotonic frame counter
FramePolicy     pol  = f.Policy();
VkFence         fnc  = f.Fence();
VkSemaphore     ia   = f.ImageAvailable();
VkSemaphore     rf   = f.RenderFinished();
VkCommandBuffer cb   = f.PrimaryCmd();    // already in BeginRecording state
JobGraph&       jobs = f.Jobs();
GpuSubmissionBatcher& subs = f.Submissions();

// record vkCmd* against cb, then:
f.QueueGraphics();                        // wires ia → rf around PrimaryCmd
// or with custom info:
GpuSubmissionBatcher::SubmitInfo info;
info.waitSem   = ia;
info.waitStage = VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT;
info.signalSem = rf;
f.QueueGraphics(info);

FrameScheduler

Owns the per-slot frame state. Composes on top of core; doesn't touch VulkanCommand / VulkanSync internals — borrows them.

Init

FrameScheduler scheduler;

FrameScheduler::Config cfg;
cfg.policy         = FramePolicy::Pipelined;
cfg.asyncMaxLag    = 2;
cfg.enableTimeline = device.HasTimelineSemaphores();
cfg.jobWorkers     = 0;   // 0 → hardware_concurrency

scheduler.Initialize(device, command, sync, cfg);

Frame loop (rule 7 — user owns Acquire / Present)

while (!window.ShouldClose()) {
    window.PollEvents();

    VCK::HandleLiveResize(window, swapchain, framebuffers, pipeline, scheduler);

    Frame& f = scheduler.BeginFrame();

    uint32_t imageIndex = 0;
    VkResult acq = vkAcquireNextImageKHR(device.GetDevice(),
                                         swapchain.GetSwapchain(),
                                         UINT64_MAX,
                                         f.ImageAvailable(),
                                         VK_NULL_HANDLE,
                                         &imageIndex);
    if (acq == VK_ERROR_OUT_OF_DATE_KHR) { /* recreate path */ continue; }

    // record into f.PrimaryCmd():
    //   vkCmdBeginRenderPass / draws / vkCmdEndRenderPass

    // Dispatch CPU jobs BEFORE queuing GPU work so they overlap
    scheduler.DispatchJobs();                            // CPU jobs run while GPU records
    f.QueueGraphics();                                   // queue the primary cmd
    scheduler.EndFrame();                                // flush batcher, advance slot

    VkPresentInfoKHR pi{ VK_STRUCTURE_TYPE_PRESENT_INFO_KHR };
    VkSemaphore rf = f.RenderFinished();
    pi.waitSemaphoreCount = 1;
    pi.pWaitSemaphores    = &rf;
    pi.swapchainCount     = 1;
    VkSwapchainKHR sc     = swapchain.GetSwapchain();
    pi.pSwapchains        = ≻
    pi.pImageIndices      = &imageIndex;
    vkQueuePresentKHR(device.GetPresentQueue(), &pi);
}

Shutdown

vkDeviceWaitIdle(device.GetDevice());     // ONLY at shutdown (rule 4)
scheduler.Shutdown();

DrainInFlight

Waits every slot's most-recent submit to retire without vkDeviceWaitIdle. Used by the scheduler-aware HandleLiveResize so dedicated compute / transfer queues keep moving across a resize event. Safe between frames; no-op while InFrame().

scheduler.DrainInFlight();              // sched waits its own slots only
swapchain.Recreate(w, h, /*drainedExternally=*/true);
framebuffers.Recreate(pipeline);

HandleLiveResize(window, swapchain, framebuffers, pipeline, scheduler) does that sequence for you.

Accessors

uint64_t              n      = scheduler.AbsoluteFrame();
uint32_t              slot   = scheduler.CurrentSlot();
FramePolicy           pol    = scheduler.Policy();
uint64_t              ret    = scheduler.LastRetiredFrame();
bool                  in     = scheduler.InFrame();

QueueSet&             qs     = scheduler.Queues();
GpuSubmissionBatcher& subs   = scheduler.Submissions();
BackpressureGovernor& gov    = scheduler.Governor();
DebugTimeline&        timl   = scheduler.Timeline();
TimelineSemaphore&    ts     = scheduler.FrameTimeline();   // valid only when enableTimeline + device support
DependencyToken       tok    = scheduler.SlotToken(slot);
const auto&           cfgRef = scheduler.Cfg();

What's next

  • VMM — the third optional layer
  • Cookbook — recipes that use the scheduler
  • ExamplesJobGraphExample, SubmissionBatchingExample, TimelineExample, SchedulerPolicyExample

VCK · Vulkan Core Kit

Getting Started

Guides

Reference

More


Single source of truth for the full API surface is the doc block at the top of VCK.h.

Clone this wiki locally