WSL2 ghost CUDA context: 1.6 GiB unrecoverable VRAM blocks 14B fp16 EvalPlus on 32 GB GPU

[joelteply]

Surfaced 2026-04-07 while running EvalPlus calibration on Qwen2.5-Coder-14B base on BigMama (RTX 5090, WSL2, Ubuntu). Tracking so we can resolve it properly later — the workaround for tonight is to calibrate against the 7B model instead.

Symptom

`torch.cuda.mem_get_info(0)` reports `free: 30.20 / 31.84 GiB` immediately after a fresh boot of every Python process, with no PyTorch or vLLM process holding any GPU memory. The 1.64 GiB delta persists after killing every `python` process via `pkill -9`. `pgrep -af python` returns nothing afterward; the ghost remains.

This is the same number every time, which is consistent with a structural reservation rather than a leaked allocation from a process we can find.

Why it matters

vLLM at `gpu_memory_utilization=0.945` requests `30.09 GiB` of budget. The 14B fp16 model loads at `27.57 GiB`. Working memory + activations consume ~2.7 GiB. Total used = `30.27 GiB`. Available KV cache memory = -0.18 GiB, which is the actual error vLLM reports.

The model can't be evaluated at fp16 on this hardware tonight without the 1.64 GiB the ghost holds. Lowering `gpu_memory_utilization` further moves the constraint from "startup memory check" to "no available memory for cache blocks", but doesn't actually free any space because the ghost is structural.

Likely cause

WSL2 GPU paravirtualization. Windows-side display driver, the WSL2 GPU passthrough layer, or DirectX cached state holds a CUDA context that the Linux-side processes cannot release. The 1.64 GiB number is consistent with a typical WSL2 GPU passthrough overhead.

`fuser -v /dev/nvidia*` requires sudo, which we don't have on BigMama tonight; running with sudo would name the holder. `nvidia-smi --gpu-reset` is the heavy hammer but also requires root.

Workarounds (in order of preference)

1. Use a smaller calibration model. Qwen2.5-Coder-7B base at fp16 = ~14 GiB, fits with 16 GiB of headroom even with the ghost. Tonight's calibration uses this. Qwen-published HumanEval = 61.6 (vs 64.0 for 14B). Calibrate eval pipeline against 7B, then trust the same pipeline for the 14B v1.5 measurement (with caveat).
2. bnb 8-bit loading via vLLM (`quantization="bitsandbytes"`). Halves model weights to ~14 GiB. Calibration delta from fp16 is small (typically <0.5 points per Qwen's own paper). Requires patching the evalplus vllm provider to pass the quantization arg through.
3. Reboot WSL2. `wsl --shutdown` from Windows side. The ghost usually clears for ~one fresh boot before the WSL2 GPU passthrough layer claims it again.
4. `nvidia-smi --gpu-reset` with sudo. Heavy hammer; recovers the full 32 GiB but requires WSL2 to allow it (usually doesn't on the GPU passthrough path).
5. Native Linux on the same hardware. Long-term fix: dual-boot or replace WSL2 with native Ubuntu on BigMama. The 1.64 GiB ghost is a known WSL2 cost; it does not exist on native Linux GPU passthrough.

What we did tonight

• Detected the ghost by comparing `torch.cuda.mem_get_info` against `pgrep python` (no holders, but 1.64 GiB used)
• Killed every python process: ghost remained
• Tried `gpu_memory_utilization` from 0.92 → 0.93 → 0.94 → 0.945 → 0.95: 0.948 is the ceiling (startup check), 0.945 passes startup but fails cache allocation
• Tried `max_num_seqs=1`, `enforce_eager=True`, `swap_space=0`: none recovered enough space
• Pivoted to base 7B for calibration anchor

Repro

```bash
ssh bigmama 'python3 -c "
import torch
free, total = torch.cuda.mem_get_info(0)
print(f"free: {free/10243:.2f} / {total/10243:.2f} GiB")
"'

Expected on a fresh BigMama: free: 30.20 / 31.84 GiB

(the gap = the ghost)

```

Resolution paths to consider

• (a) Patch the vllm provider to support `quantization="bitsandbytes"` so we can run 14B fp16 baselines via bnb 8-bit on hardware constrained by the ghost. This is a small patch and would generalize beyond this specific issue.
• (b) Switch BigMama to native Linux. Multi-day project, but the right long-term fix.
• (c) Document the ghost as a hardware-specific calibration constraint and use 7B as the canonical small-baseline anchor. Tonight's chosen path; cheap but limits us to 7B class for fp16 calibration.
• (d) Fix the WSL2 GPU passthrough layer upstream. Out of scope for us, but worth knowing if Microsoft/NVIDIA addresses it.

References:

• forge_v2_key_versions.md (BigMama hardware section)
• VALIDATED-TENSOR-SURGERY paper §4.1.1 (this is now another nested instance of "the validation pipeline has hidden constraints that bite at the worst time")