Add support for weight-based load balancing in Olla

[Mahdi-A98]

Proposed Feature:

Add a weight option for endpoints.

Weight could be applied to:

round_robin → classic weighted round-robin

least_conn → weighted least-connections

priority → within each priority tier, distribute traffic according to weight (two endpoint with same priority but different capacity)

[thushan]

Thanks for the suggestion! Would love to know more about the usecase so we can make the implementation work well.

• What's your current setup where you need weights? (eg. are you mixing GPU variations - H100 vs GB / RTX 6000 vs RTX 5090 etc)
• What other differences are you trying to balance - network routing, links?
• How are you currently working around this? (duplicating URLs, multiple instances, etc?)
• Which load balancer are you using right now as your primary?

I'm wondering whether based on your suggestion, if something like this (simplified) would work:

endpoints:
        - url: "http://h200-node-1:8000"
          name: "primary-h200"
          type: "vllm"
          priority: 100
          weight: 400

        - url: "http://h100-node-1:8000"
          name: "h100-cluster-1"
          type: "vllm"
          priority: 100
          weight: 250 

        - url: "http://h100-node-2:8000"
          name: "h100-cluster-2"
          type: "vllm"
          priority: 100
          weight: 250

        - url: "http://l40s-rack-1:8000"
          name: "inference-l40s-1"
          type: "vllm"
          priority: 100
          weight: 100

This is obviously hardware derived balancing.

[Mahdi-A98]

Thanks very much for considering this feature. In our setup, our product connects to multiple Ollama servers, and several of them run the same models. Since the servers have different hardware, capacities, and sometimes different network latencies, it would be really helpful to be able to assign a weight to each endpoint so traffic can be balanced according to their capabilities.

We are currently in the development stage. I initially tried using duplicate URLs, but Olla only accepts the last duplicate. I then set up a reverse proxy with different domains, which worked and Olla distributed traffic equally between them. I also tried the LiteLLM load balancer, which uses latency-based balancing—but this approach sends all requests to the endpoint with the lowest latency, leaving others unused. Additionally, LiteLLM doesn’t support per-endpoint configuration.

Also for our use case, having a load-balancer API (or administrative API) to dynamically add or remove endpoints would be extremely valuable.

thank you.

[Mahdi-A98]

I also want to mention that the sample configuration you provided would work well for my setup.

[moschroe]

For our use case, a combination of weights with #47 would be perfect. I work at a small company and we have not that many compute resources but would like to use what we have. An asymmetric distribution is the only fitting way to handle non-homogeneous HW.

The ideal feature set I would like to have might look like this:

• define weight per backend
• provide preferences
  • 7B models on slower HW
  • large models on newer HW
  • preferences only rank eligible hosts, so fail-over can always happen
• provide hard restrictions
  • not all machines have VRAM for large models
  • but all machines share a common storage location for downloaded files, so can "see" every available model
  • could be special case of preference score (or boolean flag hard: true), even preventing fail-over
• have olla use information about loaded models to dispatch a request
  • if a system has the requested model loaded, prefer it as target
  • if system with model is busy and has a queue, dispatch onto idle system, if available

Any logic keeping track of idle time, request queue and maximum expected queue time would need some tunable heuristics and hard limits to fall back onto well-defined behaviour.