fix(litellm): skip cost-map fetch offline, widen cold-start grace period

[zestrada]

Summary

We had issues running OSS-CRS due to non-determinism in LiteLLM startup and timeouts. This PR makes changes to the internal LiteLLM proxy's startup.

1. LiteLLM reaches out to github.com on every oss-crs run. GetModelCostMap fetches model_prices_and_context_window.json at startup, falling back to the copy bundled in the package when that fails. Under --offline this sets LITELLM_LOCAL_MODEL_COST_MAP=True to skip the fetch. Note this removes a startup dependency; it is not what caused those timeouts (see Validation).

Only under --offline, where the fetch cannot succeed and litellm falls back to the bundled map regardless — so nothing is given up. Online behavior is unchanged and live pricing is preserved. Making it unconditional would also be defensible (see User Impact) and is a one-line change if maintainers prefer it; this PR takes the narrower option.

2. The healthcheck window is too small for a cold start. First boot applies ~140 Prisma migrations from litellm-proxy-extras to an empty Postgres; the compose entrypoint override only replaces the ddtrace wrapper in docker/prod_entrypoint.sh, so those still run. The old window was start_period: 15s + 10 × 5s = 65s (:112-117), and measured cold start is right around that time (see Validation). When it expires, oss-crs-litellm-key-gen doesn't start and the whole OSS-CRS run aborts, because every CRS module gates on condition: service_healthy (:86-88).

User Impact

• User-facing change
• Internal-only change

No CLI, API, or config surface changes, and nothing to migrate. Online runs are unaffected Under --offline, pricing comes from the map bundled in the pinned image, which is what litellm already fell back to when the fetch failed. So there is no pricing change in either mode.

If maintainers prefer this unconditional

Dropping the {% if offline %} gate would make every internal-mode run use the bundled map. That was my original change, but moved to the gated version to keep online runs unchanged.

Against: online, fetching should be more accurate than a pinned map, especially if the OSS-CRS instance is not updated frequently. Pricing is used for max_budget on the CRS's LiteLLM virtual key (litellm-key-gen/main.py:49), so the map governs a cap on spending. The most dangerous case is under-pricing: an unpriced model contributes 0.0, the budget never trips, and spend is unbounded.

For: the accuracy gap on an up-to-date repo is measurable and small. Diffing the map bundled in the pinned image (litellm-database v1.94.0 / litellm_proxy_extras 0.4.79.post2, 2963 entries) against current upstream (2988) across all 73 models in defaults/litellm/default-models.yaml:

	Result
Price differences (input_cost_per_token / output_cost_per_token)	0
Models priced in upstream but not bundled	0

Across the full map rather than just our 73, the pinned image is missing 25 upstream entries and disagrees on 10 prices.

• claude-opus-5 and its bedrock/vertex/azure variants are absent from the bundled map. A litellm-config.yaml naming it would get no price and never trip llm_budget.
• gpt-5.6-luna / gpt-5.6-terra are priced 5× high in the bundled map (1e-06 vs 2e-07 input).

How stale does it get? The current pin is 8 days old, and Dependabot has bumped LITELLM_IMAGE roughly every 2-3 weeks (7 bumps since late April), so the window is ~3 weeks worst case. Over those 8 days upstream repriced 0 of the 73 models shipped in OSS-CRS; the only 2 repricings anywhere in the map were the two above, both released after the pin.

Also worth considering: the fetch's freshness is not guaranteed. It fails open - any DNS hiccup silently falls back to this same bundled map with only a log warning — so two runs of the same digest can already be priced from different sources with no way to tell which from the spend report.

Either way, a CI check that every model in a shipped config resolves to a nonzero cost in the pinned image would make this class of problem visible rather than silent could be something to consider (but not sure how updated/reliable those reference configs are for "models that are being used).

Release Note / Changelog

• I updated CHANGELOG.md ([Unreleased]) for user-facing changes
• No changelog entry needed (internal-only refactor/test/chore)

No deprecation or breaking behavior. No public signatures change; the edits are a Jinja template, one context key in renderer.py, and one test fixture. Happy to add an entry if maintainers read it the other way.

Validation

Timing measured on an offline host from the litellm service logs (postgres ready to accept connections → first /health/liveliness 200):

Run	LITELLM_LOCAL_MODEL_COST_MAP	start_period	cold start	outcome
before	off	15s	69.4s	healthcheck expired at ~65s → key-gen never ran → run aborted
after	on	90s	61.8s	healthy
after	on	90s	66.7s	healthy

Cold start is dominated by litellm import plus the Prisma migrations, which run on every boot because the DB is fresh each run - not by the cost-map fetch, which fails fast offline (the warning is logged ~5.6s after pg-ready; the remaining ~30s is between that warning and Running prisma migrate deploy). Removing the fetch saves only ~3-4s.

So the start_period bump is doing the real work here. The old 65s budget expired ~5s before litellm became healthy, and even the faster passing run came within ~3s of the cliff on an unloaded box. 90s (→ 90+50 = 140s) is what makes this robust rather than lucky.

Note the numbers above were collected on an older image carrying fewer Prisma migrations; so cold start on main today should be somewhat slower than measured, not faster.

Reproduced independently on a second (faster) host against the pinned digest, postgres pinned, fresh volume per run, on a --internal Docker network with no egress:

case	time to healthy	cost-map log lines
offline cold, flag off	36.4s	Failed to fetch remote model cost map … Temporary failure in name resolution. Falling back to local backup.
offline cold, flag on	36.4s	none — no fetch attempted

Confirms both halves: the fetch fails fast offline (worth <1s here, ~3-4s on the slower host, never ~30s), and the env var does cleanly suppress it.

Other checks:

• uv run verify — all five stages PASS: ruff check, ruff format, pyright, unit tests (591 passed, 29 deselected), libCRS unit tests (9 passed).

• Rendered the template with an internal-mode llm_context in three states and parsed each with yaml.safe_load. All valid YAML, start_period: 90s / retries: 10 in every case, and the secret-derived $$(cat /run/secrets/litellm_env_*) exports in command render byte-identically throughout — this service had no environment: key before, so there was nothing to collide with:

  context	rendered environment
  offline=True	['LITELLM_LOCAL_MODEL_COST_MAP=True']
  offline=False	absent
  key undefined	absent (pre-existing test contexts unaffected)

• Cost-map comparison reproducible with:

  docker run --rm --network none --entrypoint cat <LITELLM_IMAGE> \
    /app/.venv/lib/python3.13/site-packages/litellm/model_prices_and_context_window_backup.json
  

  diffed against raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json for every litellm_params.model in default-models.yaml, trying both the provider/model and bare-name lookup forms litellm uses.

• Migration count from ls -d .../litellm_proxy_extras/migrations/2* | wc -l in the pinned image.

Checklist

• I followed Conventional Commits
• I updated docs for behavior/config/CLI changes — no config or CLI surface changed; rationale is in the two inline comments
• I added/updated tests for behavior changes — no code path changed; the rendered-template assertion above is the meaningful check
• I considered backward compatibility and migration impact

[azchin]

Sorry for the delay, I was away traveling! I did an e2e smoke test: looks good to me, maybe add a unit test for rendering the compose with the offline flag on. Something like the following.

def test_offline_renders_local_cost_map_env(monkeypatch, tmp_path):
    _patch_renderer(monkeypatch)
    monkeypatch.setattr(renderer, "prepare_llm_context", lambda *_a, **_k: {
        "mode": "internal",
        "litellm_env_secret_files": {"OPENAI_API_KEY": str(tmp_path / "sec")},
        "litellm_config_path": str(tmp_path / "litellm-config.yaml"),
        "key_gen_request_path": str(tmp_path / "key_gen_request.yaml"),
        "secret_files": {}, "api_keys": {},
    })
    crs_compose = _make_crs_compose(tmp_path, [_make_crs(tmp_path, "crs-libfuzzer")])
    crs_compose.offline = True
    rendered, _ = _render(crs_compose, _make_target(tmp_path, has_repo=False), tmp_path)

    svc = yaml.safe_load(rendered)["services"]["oss-crs-litellm"]
    assert svc["environment"] == ["LITELLM_LOCAL_MODEL_COST_MAP=True"]
    assert svc["healthcheck"]["start_period"] == "90s"

[kusari-inspector]

Kusari Analysis Results:

❌ Kusari Inspector did not receive results for this commit in time, so the scan was not completed. Pushing a new commit will start a fresh scan. Please email support@kusari.dev if this persists.

[kusari-inspector]

Kusari PR Analysis rerun based on - 36a3009 performed at: 2026-08-17T14:39:12Z - link to updated analysis

[zestrada]

Updated PR to include tests