Skip to content

Commit 250c97c

Browse files
0bserver07claude
andcommitted
feat(providers): dev-only hand-pricing reconciler vs models.dev (Tier-2 T6)
The generator's --check guards only the *generated* fallback catalog. The small hand-maintained PRICING table Chimera actually bills had no drift guard and is the one that silently goes stale. scripts/audit_model_pricing.py reconciles PRICING against the public models.dev catalog and reports where a billed rate diverges from the first-party figure. Report-only by design: hand corrections always win over upstream, so it never rewrites a price. Exits non-zero on drift (CI-able) but is intentionally NOT wired into ci.yml. Default run is offline (committed model_catalog.py snapshot, no network) and high-signal (first-party comparison only; reseller markups are surfaced separately, --include-resellers opts in); --live fetches models.dev via the generator's stdlib urllib path; --json for machine consumption. Override convention: a new PRICING_OVERRIDES frozenset in cost.py marks the prefixes whose divergence from upstream is deliberate (GLM / DeepSeek-SKU placeholders, cross-endpoint billing nuances, local / open-weight $0 families). The audit skips them; membership does NOT affect runtime resolution (the hand table still always wins). Caught a real in-repo drift on the first run: deepseek-chat ($0.27/$1.10) and deepseek-reasoner ($0.55/$2.19) disagree with first-party models.dev ($0.14/$0.28). Dev-only: zero new runtime dependency; nothing under chimera/ imports the script. Tests reconcile over fixtures (no network) plus a network-free canary on the committed snapshot. Guide: model-catalog.md (Auditing the hand table). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent a2c5de9 commit 250c97c

5 files changed

Lines changed: 771 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,33 @@ commit receipts.
99

1010
### Added
1111

12+
- **Hand-pricing reconciler — a dev-only drift audit for the billed-price
13+
table** (Tier-2 T6): `scripts/audit_model_pricing.py` reconciles the
14+
hand-maintained `chimera.providers.cost.PRICING` table against the public
15+
models.dev catalog and reports where a billed rate has silently gone stale.
16+
It complements the generator's existing `--check`, which guards only the
17+
*generated* fallback catalog (`model_catalog.py`) — the small hand table
18+
Chimera actually bills had no such guard. **Report-only by design**: hand
19+
corrections always win over upstream, so it never rewrites a price; it exits
20+
non-zero on drift (CI-able) but is intentionally **not** wired into CI. The
21+
default run is offline (reconciles against the committed `model_catalog.py`
22+
snapshot, no network) and high-signal — it compares only against
23+
**first-party** manufacturer figures, since a hand rate disagreeing with a
24+
reseller's markup is margin, not drift; reseller-only ids are surfaced
25+
separately (`--include-resellers` opts in). `--live` fetches models.dev via
26+
the generator's stdlib `urllib` path; `--json` emits a machine-readable
27+
report. The **override convention** is a new `PRICING_OVERRIDES` frozenset in
28+
`cost.py` marking prefixes whose divergence from upstream is deliberate
29+
(GLM / DeepSeek-SKU placeholders, cross-endpoint billing nuances, local /
30+
open-weight `$0` families) — the audit skips them, and membership does **not**
31+
affect runtime resolution (the hand table still always wins). It caught a real
32+
in-repo drift on the first run: the DeepSeek hand rates (`deepseek-chat`
33+
$0.27/$1.10, `deepseek-reasoner` $0.55/$2.19) disagree with the first-party
34+
models.dev figure ($0.14/$0.28). Dev-only: **zero new runtime dependency**,
35+
and nothing under `chimera/` imports the script. Guide:
36+
`site/src/content/docs/guides/model-catalog.md` (new *Auditing the hand table*
37+
section).
38+
1239
- **Inline mode — the single-agent transcript in the terminal's native
1340
scrollback** (R-VIEW-5, opt-in): `chimera code --tui --inline` (or
1441
`[tui] inline = true`) renders the daily driver's committed transcript into

chimera/providers/cost.py

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,36 @@
106106
"gemma3-27b-instruct": (0.0, 0.0),
107107
}
108108

109+
# Prefixes in :data:`PRICING` whose hand rate DELIBERATELY diverges from the
110+
# public models.dev figure and must never be "corrected" toward upstream. Each
111+
# is a conscious override, for one of three reasons documented inline above:
112+
#
113+
# * a **placeholder** pending a vendor's public rate sheet (the GLM family,
114+
# the DeepSeek V3.1/coder/V4 SKUs, the Kimi preview line);
115+
# * a **cross-endpoint billing nuance** upstream can't express (a model whose
116+
# true billing depends on which endpoint or local bridge served it);
117+
# * a **local / open-weight** family billed at ``$0`` because the serving
118+
# daemon surfaces no price field.
119+
#
120+
# This is the marker convention consumed by the dev-only reconciler
121+
# ``scripts/audit_model_pricing.py``: it skips these prefixes so an intentional
122+
# divergence is never reported as drift, while still flagging a hand rate that
123+
# has silently gone stale. Membership does **not** affect runtime resolution —
124+
# :func:`get_model_pricing` always prefers the hand table regardless; hand
125+
# corrections always win. Removing a prefix here re-arms the audit for it.
126+
PRICING_OVERRIDES: frozenset[str] = frozenset({
127+
# GLM — placeholders + z.ai-vs-Ollama-bridge billing nuance.
128+
"glm-5.2", "glm-5.1", "glm-5", "glm-4.6", "glm-4-plus", "glm-4-flash",
129+
# DeepSeek V3.1 / coder / V4 — explicit placeholders pending per-SKU rates.
130+
"deepseek-v3.1-terminus", "deepseek-coder-v3",
131+
"deepseek-v4-pro:cloud", "deepseek-v4-pro", "deepseek-v4",
132+
# Kimi (Moonshot) — $0.6/$2.5 placeholder pending per-SKU rates.
133+
"kimi-k2-0905-preview", "kimi-k2.5",
134+
# Local / open-weight — billed $0 (no price field on the serving daemon).
135+
"qwen3-coder-30b", "qwen3-coder", "qwen3-32b",
136+
"gpt-oss-120b", "gpt-oss-20b", "mistral-codestral-2511", "gemma3-27b-instruct",
137+
})
138+
109139
_pricing_lock = threading.Lock()
110140

111141
# --- Generated-catalog fallback -------------------------------------------

0 commit comments

Comments
 (0)