llmfit mis-evaluating model fit memory?

[jdougan]

I don't understand how llmfit evaluates if a given model fits. Qwen3.5-9b in Q4_K_M (unsloth, model file is 5.29 GB (5,680,529,408 bytes)) is running just fine on my laptop GPU. But according to llmfit, it shouldn't. It is slow (10 tok/sec, but that is untuned.) If you look at the hardware in the screenshot, with 32GB of main ram and 8 GB of VRAM it clearly exceeds the requirement in the Notes section. I'm also unclear where it gets 26.3 GB from in the Memory section.

Maybe a bug? Or am I just confused and it is explained somewhere I missed?

[AlexsJones]

Hey, good catch! You're not confused, this is a real usability issue.

The 26.3GB number comes from estimating memory at the model's full 262K context window. The KV cache at that length is ~20GB on its own. In reality, llama.cpp/Ollama default to way smaller contexts (2048-8192), which is why it runs fine on your 8GB GPU.

For now you can get a more accurate estimate with llmfit --max-context 8192 (or whatever context you're actually using), or set OLLAMA_CONTEXT_LENGTH=8192 as an env var.

We should probably default to a reasonable cap instead of assuming full context, especially for models advertising 256K+ windows that nobody runs at full length on consumer hardware. Going to look into fixing that default.

[jdougan]

I think the calculations just need to be more explicit. Having the max context memory size is useful. Running the calculation backwards to get a max reasonable context length could be more useful: "Will not fit at full context, will fit at YYYY context with FP16 entries, ZZZZ with Q8_0 entries." etc. And if all the the context lengths have to be negative to make fit, then it really doesn't fit. If some are, then it qualifies as marginal.

BTW, you're currently assuming the KV cache is FP16?

Just reading my current config and the logs, with a full context (262144) and Q8_0 entries the KV cache at startup is around 4GB. I'm new to this, I'm not sure if it grows later or not.

[jdougan]

And after poking at llama.cpp some more, the --fit on option will adjust the number of layers on the GPU if you insist you want a big context... Seems like lots of potential parameter fiddling. I wonder if you could steal some of llama.cpp's fitting logic for this?