Skip to content

v0.3.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 03 Jun 23:44
v0.3.0
96fd1fe
Changelog
  • Updated to a much newer llama.cpp base. The upstream refresh brings native MTP speculative decoding, parallel drafting and backend sampling work, the unified llama app, newer server/API behavior, the tools/ui Web UI restructure, model and converter additions, multimodal improvements, and backend gains across CUDA/HIP, Metal, Vulkan, SYCL, OpenCL, WebGPU, Hexagon, ZenDNN, and SpacemiT.
  • Added usable MTP serving on the Bee tree. draft-mtp/mtp now has its own speculative path instead of being mixed with DFlash state, uses draft KV cache types, cleans up draft resources on sleep, works with target pre-norm hidden-state capture, and handles text requests when an mmproj is loaded.
  • Cleaned up DFlash command-line behavior around canonical --spec-* arguments. DFlash can still be selected with --spec-type dflash or auto-detected from compatible draft GGUF metadata, but stale aliases such as --spec-dflash-default, --draft*, --draft-topk, --draft-model, --tree-budget, --dflash-max-slots, --spec-draft-replace, and --spec-replace were removed.
  • Made DFlash defaults match the new upstream speculative surface without changing the practical DFlash defaults. Raw --spec-draft-n-max stays at upstream 3, DFlash raises the effective omitted draft max to 16 only after explicit or auto-detected DFlash, omitted --spec-draft-ctx-size still becomes 256, and DFlash no longer lowers target -b 2048 / -ub 512 unless the user asks for smaller batches.
  • Made DFlash-only arguments safe around non-DFlash modes. DFlash-only controls warn and no-op for MTP or other speculative types, while still surviving long enough for server-side DFlash draft-model auto-detection when no explicit --spec-type dflash was passed.
  • Expanded multi-slot DFlash serving. DFlash slots now default to server parallelism, --spec-dflash-max-slots caps them below -np when needed, uneven -np target batches are split safely, mixed speculative/non-speculative target batches are avoided, and flat multi-slot DFlash can use shared drafter batching with GGML_DFLASH_SHARED_DRAFT_BATCH=0 as the fallback switch.
  • Reworked adaptive DFlash draft depth for live serving. The default profit controller now seeds and periodically remeasures a no-spec baseline, probes shallow/mid/full positive depths, backs off failed wake probes, avoids premature demotion, resets per-request state correctly, preserves compatible continuation state, and gates timing logs behind DFlash profiling.
  • Added default-on device-aware DFlash GPU capture/tape/replay for split CUDA/ROCm target placement. Hidden capture, prefill capture, recurrent tape, conv replay, direct GDN replay, rollback copies, and synchronization now follow each layer's backend device, with CPU/eval-callback fallback and GGML_DFLASH_MULTI_GPU_TAPE=0 / GGML_DFLASH_ALLOW_MULTI_GPU_TAPE=0 kill switches.
  • Hardened DFlash device placement. Explicit single draft-device placement pins the target output tensor before target load, auto-detected DFlash drafters stay single-device by default unless draft devices are explicit, iGPU backends are accepted for GPU paths, tensor-split/meta placement is guarded, CUDA helper calls preserve the caller device, and peer D2D copies are used where available.
  • Hardened DFlash hidden-ring and prompt-cache state. The server guards GPU hidden-ring spans, discards stale DFlash ring checkpoints, skips DFlash checkpoints for uncached prompts, restores shared prefill capture state, and avoids requesting raw prompt logits from DFlash paths that should not read them.
  • Reduced DFlash accept and verification overhead while failing closed on bad reduced-logit state. The accept path defers single-slot rollback sync and drafter KV maintenance, keeps the fused GDN 4D state fast path, adds CPU f16 out_prod fallback, and disables DFlash drafting after repeated invalid reduced-logits drafts instead of looping or corrupting state.
  • Improved DFlash tool-call and reasoning behavior. DFlash can keep drafting before a lazy tool-call grammar actually constrains output, stale drafter KV state is cleared before long tool-call continuations, stable partial tool-call headers stream earlier, streaming reasoning deltas stay isolated, and streamed title generation suppresses leading thinking syntax.
  • Kept flat DFlash available with multimodal serving while making unsupported combinations explicit. With --mmproj, Bee keeps flat DFlash usable, forces DFlash tree branch budget to 0, disables non-DFlash speculative modes, disables unsupported context-shift/cache-reuse paths, and fixes MTP text-only requests when an mmproj is present.
  • Updated Qwen 3.5/3.6 speculative paths. Qwen gets per-layer KV heads, final-layer output gathering, stable MTP draft context behavior, and Qwen DDTree conv/GDN paths for tree verification.
  • Removed legacy DDTree total-node semantics from the public CLI. Tree DFlash now uses the branch-only --spec-branch-budget model consistently, with --spec-draft-top-k controlling candidates per draft position and flat DFlash forcing top-k back to 1.
  • Added new cache and quantization surface beyond v0.2.0. q6_0 is available as a KV/cache type, and Tom's TQ3_1S / TQ4_1S model weight formats are exposed through llama-quantize with non-conflicting serialized GGML type IDs; existing turbo2, turbo3, turbo4, turbo2_tcq, and turbo3_tcq cache types remain available on the newer base.
  • Hardened backend behavior used by Bee features. HIP TCQ attention stays on the native vector path, ROCm can probe fused GDN support, D512 Flash Attention selection is available again, unsupported CPU BF16 scale and CPU Flash Attention types are rejected instead of misrunning, CUDA fatal-warning cases were fixed, and the build now requires C++17 for the common base.
  • Improved server and API behavior inherited from upstream and Bee integration. The server reports prompt token counts in /slots, supports SSE ping intervals and HTTP ETags, exposes real-time reasoning interruption, handles router/model metadata fixes, adds built-in tools such as datetime, and keeps malformed tool-looking text out of final responses.
  • Updated packaging and release workflows. Docker image links and labels were aligned for BeeLlama, SYCL package/release images were added and documented, release builds were split and cached more reliably, stale package builds are cancelled, and obsolete CUDA architecture options are rejected at configure time.
  • Updated user documentation for the new release state, including DFlash args/defaults, removed aliases, adaptive Draft-Max, multi-GPU DFlash behavior, Turbo/TQ cache and weight formats, Docker images, and the upstream multi-GPU guide.
  • Expanded regression coverage around DFlash default normalization, removed arg aliases, DFlash-only no-op behavior, DFlash auto-detection, multi-GPU policy helpers, per-layer capture/tape allocation, device-aware replay, CUDA device restoration, adaptive Draft-Max, reduced-logit failure handling, and tool-call/speculative boundaries.

macOS:

Linux:

Windows:

Docker:

  • CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.3.0
  • CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.3.0
  • CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.3.0
  • CUDA 13: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.3.0
  • ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.3.0
  • Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.3.0
  • SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.3.0

Browse all container images

Assets 15
Loading