Memory calculator

[jeremyfowers]

The biggest user friction we have right now is making sure a good model with the right settings (i.e., recipe) is loaded for agentic workloads like OpenClaw and Claude Code.

We can't automatically choose a recipe for users because we don't know what will cause them to OOM. Knowing this requires a memory calculator that takes into account the model size, and how that model's architecture changes the RAM requirement with critical settings like ctx_size.

One way to solve this is brute force: load all the top tool-calling models with varying context sizes, measure memory pressure, put the data in a table, and do a lookup on that table at runtime.

Alternatively, we could try to come up with a formula for RAM usage. This might be tricky though because its highly model-architecture and implementation-dependent, and can change across llamacpp releases.

Ideas welcome!

cc @sawansri

[fl0rianr]

Hello 🙂

A practical middle ground might be to treat memory usage as something Lemonade learns at runtime, rather than trying to predict it perfectly up front.

For example, on the first successful model load, Lemonade could record start, peak, and steady-state RAM usage for that specific recipe and cache it locally. This would likely be more reliable than a pure formula, since memory usage depends on model architecture, ctx size, backend, and even llama.cpp version.

A “recalculate profile” option would also be useful, since memory behavior can change across releases. Optionally, profiles could be refreshed automatically from time to time.

It might also help to include a basic sanity check against estimated usage, and surface large deviations to the user (e.g. suggesting config adjustments).

Additionally, supporting memory budgeting directly (e.g. “free 30 GB”) instead of only unloading named models could improve usability, especially for automation workflows where the caller does not know the actual RAM usage of loaded models.

Overall, this could reduce a lot of friction around recipe selection and OOM issues.

[jeremyfowers]

@fl0rianr thanks for the suggestion! That is something we could do down the line.

The primary problem I'm trying to solve right now though is a first time user comes in and tries to launch Claude Code or OpenClaw with too small of a context size. We need to load a good context size for them on this very first try, without any historical data, or else they have a bad first experience with Lemonade.

[fl0rianr]

Thanks for clarifying — I think the most practical first-time solution might be a probe load + fast resize approach.

The basic idea would be:

1. load the model once with a small, known-safe default context
2. observe the real memory usage on that machine
3. if there is enough headroom, reload with a better default ctx_size
4. cache that result for future launches

The important part is that this should ideally be split into model load and context load. That way Lemonade would not have to fully reload the model weights just to adjust context size. Instead, it could keep the model resident, do a small safe probe context first, and only recreate the context with a larger default if the measured headroom allows it.

Pros:

• safe first launch
• better first-time defaults
• no need for a perfect formula up front
• improves quickly based on the actual machine / backend combination

I would probably also only do the second load if the gain is meaningful, so first-time UX stays fast.

[jeremyfowers]

The important part is that this should ideally be split into model load and context load.

@fl0rianr does llamacpp support that?

[fl0rianr]

Good morning @jeremyfowers 🙂

Good news is, at the llama.cpp API level, model and context are already separate.

In include/llama.h:

• the model is loaded separately via llama_model_load_from_splits(...)
• the context is then created from that loaded model via llama_init_from_model(model, params)
• and they are also freed separately via llama_model_free(...) and llama_free(...)

n_ctx also lives in llama_context_params, so it is clearly part of context creation rather than model loading.

If this direction is interesting, I could try to put together a small test / prototype around this (just to validate the probe-load idea in practice).

Also, are you generally open to more lightweight / irregular contributions? I don’t have the capacity to be a maintainer, but I’d be happy to contribute ideas or small experiments here and there — I think the project is really promising (and I have a gfx1151 system at hand).

Happy to explore this further if useful.

[jeremyfowers]

Sorry about the slow reply @fl0rianr!

Yes a prototype of this would definitely be interesting.

Also, are you generally open to more lightweight / irregular contributions?

Yeah definitely. Just need to make sure someone is available to maintain the feature before merging it.