submissions: gemma-4-26B-A4B MoE-CPU-offload on Predator GTX 1060 6GB - #1
Open
slobodanmargetic988 wants to merge 1 commit into
Open
submissions: gemma-4-26B-A4B MoE-CPU-offload on Predator GTX 1060 6GB#1slobodanmargetic988 wants to merge 1 commit into
slobodanmargetic988 wants to merge 1 commit into
Conversation
…GTX 1060 6GB Off-canonical run (single-call latency probe per quant, NOT the public-1 5q/20q chat suite) testing whether `--override-tensor "exps=CPU"` makes the 26B/A4B Gemma 4 MoE serve viably on a Pascal 6GB GPU when neither the IQ2 nor Q4 dense form fits in VRAM. Two cells, both `predator:llamacpp:gemma-4-26b-a4b-it-*`: - IQ2_XXS (9.24 GB on disk): 10.01 t/s decode, 4012 MB VRAM - Q4_K_M (15.78 GB on disk): 9.03 t/s decode, 5823 MB VRAM (200 MB free) Pattern works on Vulkan + Pascal. Decode rate is dominated by CPU-expert memory bandwidth, so the bigger Q4 quant pays only a 10% speed cost for materially better quality. ~2.6× slower than the dense gemma-4 E4B run on the same hardware (run 09d8fbde) in exchange for 6.5× the total weight count at similar active-param compute. Caveats called out in run.md: - n=1 per cell; treat as feasibility data, not steady-state throughput - gemma-4 thinking-mode swallowed the 200-token user-visible output (timing numbers still valid — 200 generated tokens is real work) - shared-host baseline (other supervised services running but idle) Should grade `archive-only` or `standard` at most — flagship requires the canonical 5q+20q methodology which this run intentionally skipped.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d223d95720
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ], | ||
| "storage": {"kind": "ssd", "free_gb_at_run_start": 637}, | ||
| "thermal_or_power_notes": "On AC power; supervisor-managed services running concurrently (gemma-4 E4B llama-server on 11437, ollama on 11434, weeyuga serve on 11435) but idle during the bench window. Test llama-server on 11439.", | ||
| "network_used_for_model_fetch": "WireGuard 10.8.0.7 mesh + residential 100mbps internet for HF download (only used to pull GGUFs to D: before the bench; not on the timing path)", |
There was a problem hiding this comment.
The submission includes a specific WireGuard private address (10.8.0.7) in network_used_for_model_fetch, which exposes internal network topology and conflicts with the repo's privacy guardrails for submitted runs. This field can still document connectivity (e.g., "WireGuard mesh") without publishing an exact internal IP.
Useful? React with 👍 / 👎.
| @@ -0,0 +1,4 @@ | |||
| {"type":"meta","benchmark_run_id":"2dc8fd67-8c53-4980-909e-d94b8aea926c","harness_version":"ad-hoc-moe-cpu-offload-2026-05-09","started_at_utc":"2026-05-09T07:35:00Z","host_hostname_short":"DESKTOP-02M21DO","load_avg_start":null,"target_url":"http://127.0.0.1:11439","cell_id_prefix":"predator:llamacpp","submitter_handle":"slobodanmargetic988","device_tag":"gtx-1060-6gb","execution_shape":"single-call-cold-per-quant","phases_planned":["single-call-cold"],"models_planned":["gemma-4-26B-A4B-it-UD-IQ2_XXS.gguf","gemma-4-26B-A4B-it-UD-Q4_K_M.gguf"],"canonical_options":{"temperature":0.0,"max_tokens":200,"ctx_size":4096,"ngl":99,"override_tensor":"exps=CPU","threads":4},"canonical_options_effective":{"temperature":0.0,"max_tokens":200,"ctx_size":4096,"ngl":99,"override_tensor":"exps=CPU","threads":4},"timeout_seconds":600,"platform_system":"Windows","platform_release":"10","python_version":"3.10.6","note":"NOT a public-1 canonical 5q/20q run; this is a focused MoE-CPU-offload feasibility bench. See run.md."} | |||
| {"type":"call","ts_utc":"2026-05-09T07:42:00Z","cell_id":"predator:llamacpp:gemma-4-26b-a4b-it-iq2_xxs","model":"gemma-4-26B-A4B-it-UD-IQ2_XXS.gguf","phase":"single-call-cold","question_id":"transformer_explainer","run_idx":0,"duration_seconds":22.77,"prompt_tokens":34,"completion_tokens":200,"tokens_per_second":8.78,"finish_reason":"length","status_code":200,"response_chars":null,"response_preview":"<thinking-mode-output-redacted: gemma-4 in default thinking mode emitted reasoning tokens that filled the 200-token max; user-visible content was empty>","required_markers":[],"markers_hit":[],"marker_hit_rate":null,"format_rule":"","format_ok":null,"usable_answer":null,"error":null,"server_timings":{"prompt_per_second":13.25,"prompt_n":34,"prompt_ms":2566,"predicted_per_second":10.01,"predicted_n":200,"predicted_ms":19980},"vram_mb_used":4012,"vram_mb_total":6030} | |||
There was a problem hiding this comment.
This call record stores a non-empty placeholder string in response_preview while response_chars is null and run.md states the preview is null for both calls. That inconsistency makes the run self-contradictory and can skew downstream consumers that treat non-null previews as actual model output; use null for response_preview when no user-visible text was produced.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds one off-canonical submission under
submissions/slobodanmargetic988/gtx-1060-6gb/:--override-tensor "exps=CPU"MoE pattern on llama.cpp Vulkan + Pascal, using the newunsloth/gemma-4-26B-A4B-it-GGUFquantspredator:llamacpp:gemma-4-26b-a4b-it-iq2_xxs(9.24 GB on disk) → 10.01 t/s decode, 4012 MB VRAMpredator:llamacpp:gemma-4-26b-a4b-it-q4_k_m(15.78 GB on disk) → 9.03 t/s decode, 5823 MB VRAM (only 207 MB free at ctx=4096)Pattern is viable. Decode rate dominated by CPU-expert memory bandwidth, so Q4 vs Q2 only costs ~10% speed for a real quality jump. ~2.6× slower than the dense gemma-4 E4B run on the same Predator (
runs/09d8fbde-...) in exchange for 6.5× the total weight count at similar active-param compute.Methodology call-out
⚠ This is NOT the canonical public-1 5q+20q chat suite. It's a single-call latency probe per quant (
max_tokens=200, one prompt). The CONTRIBUTING.mdcaveat_severityfor this should bewarningand gradearchive-onlyorstandardat most —flagshiprequires the canonical methodology.The off-canonical shape is justified in
run.md§"Why this off-canonical shape": the question this run answers (does the pattern work and at roughly what speed?) is one-call-per-quant; sustained-throughput / format-correctness is a different question that needs a different run.Caveats (also in run.md)
response_previewis null in run.jsonl because gemma-4-it's default thinking-mode consumed the full 200-token budget before any user-visible content emerged. Timing numbers are still valid (200 tokens generated either way); format-correctness scoring would need highermax_tokensor--reasoning-budget 0Test plan
manifest.jsonvalid JSON — checkedhardware.jsonvalid JSON — checkedmetadata.jsonvalid JSON — checkedrun.jsonlvalid (meta → 2× call → footer) — checked/Users//homepaths — onlyD:\Windows install paths describing the hardware setupsubmissions/EXAMPLE/not modifiedWhat this validates for the cluster
gemma-4-26b-a4b-it-llamacpp-vulkan-q4_k_mis now in Predator's~/.weeyuga/models.yamlregistry as an on-demand entry (not a permanent supervisor slot yet). When the picker dispatches to it, the user is expected to spin up llama-server with the documented args via llauncher first; the registration makes it visible in Settings → Staff. Promoting it to a full supervisor slot is a separate decision.🤖 Generated with Claude Code