Skip to content

Commit 445e5a4

Browse files
JonathanBechtelJonathanBechtel
committed
docs(methodology): add harness review findings + precision/judge-scope/contamination disclosures
Adds docs/methodology_review_findings.md — the full pre-launch methodology review of the evaluation harness (14 findings across critical/high/medium with fix status and file references) — and extends docs/methodology.md with three disclosure sections it calls for: - Statistical precision: 26 tasks, per-track N as low as 5; per-track gaps within plausible noise - Judge scope: the LLM judge scores against gold material, not raw fixture CSVs; numbers are verified by the deterministic C1 scorer - Benchmark contamination: fixtures regenerate per numbered benchmark version; scores comparable only within a grade_version Also documents the cross-family judge-selection policy in the C2 scoring bullet (Opus 4.8 judges all non-Claude candidates; GPT-5.5 xhigh judges Claude candidates) with its two residual caveats.
1 parent b657d47 commit 445e5a4

2 files changed

Lines changed: 253 additions & 1 deletion

File tree

‎docs/methodology.md‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,15 @@ The scoring pipeline:
186186
- **C1 (fact scoring)** — compares structured numeric outputs against `gold_facts` with
187187
absolute or relative tolerance.
188188
- **C2 (rubric scoring)** — a judge model evaluates each of the six dimensions using
189-
task-authored per-dimension guidance.
189+
task-authored per-dimension guidance. Judge selection is **cross-family**: Claude
190+
Opus 4.8 judges every candidate except Claude-family models, which are judged by
191+
GPT-5.5 at xhigh reasoning effort instead — no judge ever shares a model family
192+
(and hence a house style) with the model it grades. Two caveats are inherent to
193+
this design and disclosed here: (1) both judges are themselves benchmark
194+
contestants, though neither ever judges its own family; (2) Claude rows are scored
195+
by a different judge than all other rows, so a systematic harshness difference
196+
between the two judges would shift the judged dimensions (40% of the composite)
197+
for those rows relative to the rest of the leaderboard.
190198
- **C3 (claim validation)** — deterministic string-matching (with optional judge fallback)
191199
checks `gold_insights` presence, `forbidden_claims` absence, and `required_limitations`
192200
presence.
@@ -227,5 +235,26 @@ response consistency for education program analytics tasks using realistic synth
227235
are validated in `ground_truth.json`; a different seed produces different realized values
228236
that may shift task difficulty.
229237

238+
**Statistical precision.** GRADE V1 contains 26 tasks (11 operations, 5 outcomes,
239+
10 equity & research), with per-track task counts as low as 5. Overall composites are
240+
averaged over 26 tasks × 5 runs; per-track and per-pack scores rest on much smaller
241+
samples and should be read as indicative rather than precise. Small leaderboard gaps
242+
(roughly a few points) on per-track views are within plausible noise.
243+
244+
**Judge scope.** The LLM judge evaluates the judged dimensions against the task's gold
245+
material (gold facts, gold insights, required limitations, forbidden claims, reference
246+
outline) — it does **not** see the raw fixture CSVs. Verifying numbers against the data
247+
is the deterministic C1 scorer's job; the judge assesses interpretation quality relative
248+
to the gold material. This keeps judge prompts bounded and judging reproducible, at the
249+
cost that `evidence_linkage` is assessed on plausibility against gold material rather
250+
than re-derived from the data.
251+
252+
**Benchmark contamination.** Fixture data, gold facts, and task prompts are public in
253+
this repository, so future model training runs may ingest them. Because every dataset is
254+
produced by the synthetic generator (`benchmark/datagen/`), GRADE mitigates this by
255+
regenerating fixtures (new seed, recomputed `ground_truth.json`) for each numbered
256+
benchmark version; scores are only comparable within a benchmark version, and the
257+
version is recorded in every result (`grade_version`).
258+
230259
The public/private boundary governing what can be published is documented in
231260
[`docs/publication_policy.md`](publication_policy.md).

‎docs/methodology_review_findings.md‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# GRADE Methodology Review — Findings & Fix Plan
2+
3+
**Date:** 2026-06-10
4+
**Scope:** Full review of the evaluation harness (`runner/`, `benchmark/rubrics/`, `scripts/run_models.py`) ahead of the first real leaderboard run (issue PearlEng/platform-ai#724).
5+
**Status legend:** `FIXED` (this session) · `OPEN` (assigned to follow-up session) · `DISCLOSE` (methodology-page disclosure, not a code change)
6+
7+
---
8+
9+
## Critical (launch-blocking)
10+
11+
### C-1. Model registry mislabels which models actually run — `FIXED`
12+
`SEED_MODELS` in `runner/adapters/openrouter_adapter.py` mapped shorthands to
13+
stale placeholder slugs: `"gpt-5"` → `openai/gpt-4o`, `"gemini-2.5-pro"` →
14+
`google/gemini-pro-1.5`, `"claude-opus-4-7"` → `anthropic/claude-opus-4.5`.
15+
A leaderboard published from these would attribute GPT-4o scores to GPT-5.
16+
The adapter's default model and the judge default also used dash-form slugs
17+
(`anthropic/claude-sonnet-4-5`) that do not exist on OpenRouter (the live
18+
slugs use dots, e.g. `anthropic/claude-sonnet-4.6`).
19+
20+
**Fix applied:** `SEED_MODELS` replaced with the verified launch lineup (see
21+
"Launch model lineup" below; every slug checked against the live
22+
`/api/v1/models` endpoint on 2026-06-10). Adapter default model and judge
23+
default corrected to live slugs.
24+
**Follow-up (separate repo):** `grade-app/arena/pairs.py`
25+
`SEED_MODEL_DISPLAY_NAMES` must be updated to match the new lineup when real
26+
results are imported (part of #724).
27+
28+
### C-2. Forgetting `--judge` silently produces a fake leaderboard — `FIXED`
29+
Without a judge client, `insight_quality`, `evidence_linkage`, and
30+
`structure_usability` (40% of the composite) were scored as a flat `0.5` for
31+
every model by `_NullJudge`, and nothing in the result recorded this. A
32+
misconfigured run would yield a plausible-looking leaderboard ranked only by
33+
fact-matching.
34+
35+
**Fix applied:**
36+
- `runner/dispatcher.py` now stamps a `null_judge` flag into every task's
37+
`scorer_flags` when no live judge is configured.
38+
- `runner/cli.py` and `scripts/run_models.py` now **refuse to run** the
39+
`openrouter` adapter without `--judge` unless `--allow-null-judge` is
40+
passed explicitly (stub adapter unaffected; smoke tests keep working).
41+
42+
### C-3. Judge reliability: invalid slug, silent zeros, no retry — `FIXED` (partially `DISCLOSE`)
43+
`benchmark/rubrics/judge_client.py`:
44+
- `DEFAULT_JUDGE_MODEL` was `anthropic/claude-opus-4-5` — an invalid slug
45+
(live slug is `anthropic/claude-opus-4.5`), so every live judge call would
46+
have failed.
47+
- A judge reply that didn't parse as a float silently became `0.0` — a judge
48+
hiccup charged against the candidate model, with no log, retry, or flag.
49+
50+
**Fix applied:**
51+
- Default judge slug corrected to a live dot-form slug (now
52+
`anthropic/claude-opus-4.8` per the selection policy below).
53+
- Response parsing now extracts the first float via regex (tolerates stray
54+
punctuation/markdown), retries the API call once on parse failure, and
55+
raises `JudgeScoreError` if both attempts fail — so the failure surfaces in
56+
`failures.json` instead of silently zeroing a dimension.
57+
- `--judge-model` flag added to both CLIs so the judge is configurable per
58+
run.
59+
60+
**Judge selection policy (decided 2026-06-10, revised same day, implemented):**
61+
the strongest available model judges everyone — the principle being that a
62+
weaker model should not grade a stronger one — and **no judge ever shares a
63+
model family with its candidate**, so house style cannot bias the judged
64+
dimensions. Default judge is **Claude Opus 4.8**; every **Claude-family
65+
candidate** (Opus, Sonnet, Haiku) is instead judged by **GPT-5.5 at xhigh
66+
reasoning effort**. Implemented in
67+
`benchmark.rubrics.judge_client.select_judge_model()` and applied
68+
automatically by both CLIs (`--judge-model` overrides). Reasoning-effort
69+
passthrough was added to `post_chat_completion` for this (the GPT-5.5 judge
70+
sends `reasoning: {effort: "xhigh"}` with an 8192-token budget, since
71+
reasoning tokens share the budget with the one-float answer).
72+
73+
**`DISCLOSE` — two residual caveats for the methodology page:**
74+
1. *Contestant judges*: both judges are themselves contestants (Opus 4.8 is
75+
judged by GPT-5.5; GPT-5.5 rows are judged by Opus 4.8). Neither ever
76+
judges its own family, but neither is a disinterested third party either.
77+
2. *Calibration mismatch*: Claude rows are scored by a different judge than
78+
all other rows. If the two judges differ in harshness, the Claude rows
79+
sit on a slightly different scale for the three judged dimensions (40% of
80+
composite). Mitigation if it proves material: run both judges over a
81+
small overlap sample and report (or correct for) the judge-to-judge
82+
offset.
83+
84+
---
85+
86+
## High (affects score validity — fix before the real run)
87+
88+
### H-1. Grounding accuracy is credit-only and key-agnostic → rewards number spam — `OPEN`
89+
`benchmark/rubrics/fact_scoring.py`: a gold fact is credited if **any**
90+
number anywhere in the output (up to 50 prose sentences, plus a `/100`
91+
percent-normalized variant of every candidate) falls within tolerance.
92+
Leniency floors widen the window further (gold count 30 with authored
93+
tolerance 0 gets ±0.6, so an unrelated "29.6%" anywhere credits it). There is
94+
no penalty for wrong/hallucinated numbers. Net effect: verbose, number-dense
95+
outputs systematically outscore concise ones on the highest-weighted
96+
dimension (0.35–0.40).
97+
98+
**Suggested fix:** require the matching number to appear in a sentence with
99+
token overlap against the gold claim (or within N tokens of a claim keyword),
100+
and/or add a precision-style penalty for confidently asserted numbers that
101+
match no gold fact. At minimum, log per-fact `method` labels into the result
102+
for auditability.
103+
104+
### H-2. Non-numeric gold facts effectively require verbatim echo — `OPEN`
105+
`fact_scoring.py:score_fact` finds a candidate finding by substring
106+
containment but then scores it with **exact** string equality
107+
(`score_exact_match`), so a finding that contains the claim plus any other
108+
words scores 0. The module docstring promises token-overlap matching that is
109+
not implemented. Non-numeric facts are near-universal misses for all models.
110+
111+
**Suggested fix:** score the substring/token-overlap match directly (like
112+
C3's `TOKEN_OVERLAP_THRESHOLD` approach) instead of exact equality.
113+
114+
### H-3. `runs=1` gives every model a free 10% (consistency) — `OPEN`
115+
All three consistency sub-metrics default to 1.0 with a single run. The
116+
leaderboard run must use `--runs ≥ 3` (default 5 is good). Suggested fix:
117+
stamp a `consistency_trivial` scorer flag when `runs < 2`, or exclude the
118+
dimension from the composite in that case.
119+
120+
### H-4. `max_tokens=1024` truncates analyses; reasoning models break — `OPEN`
121+
`OpenRouterAdapter.DEFAULT_MAX_TOKENS = 1024` is tight for a multi-section
122+
prose analysis; limitations sections come last and get cut first (deflating
123+
`calibration_limitation_handling`). `finish_reason` is not checked, so
124+
truncation is invisible. Reasoning models (GPT-5.5 at high effort) can burn
125+
the entire budget on reasoning tokens and return an empty visible response.
126+
127+
**Suggested fix:** raise the default to ≥ 4096, check
128+
`choices[0].finish_reason == "length"` and stamp a `truncated` flag, and add
129+
a `reasoning` parameter passthrough (see H-5).
130+
131+
### H-5. No reasoning-effort support — required for the GPT-5.5 sweep — `OPEN` (partial)
132+
The launch plan includes GPT-5.5 at xhigh/high/medium/low effort.
133+
**Done (2026-06-10):** `post_chat_completion` now accepts a
134+
`reasoning_effort` parameter (used by the GPT-5.5 judge).
135+
**Still open:** `OpenRouterAdapter` (the candidate-side path) does not yet
136+
accept/forward a reasoning effort, and results need distinct `model_id`
137+
labels per effort level (e.g. `openai/gpt-5.5@xhigh`) so leaderboard rows
138+
don't collide. Wire the adapter through the same parameter and add a
139+
`--reasoning-effort` CLI flag (or per-model syntax in `--models`).
140+
141+
---
142+
143+
## Medium (quality / cost — assigned to follow-up session)
144+
145+
### M-1. C1 never sees the `limitations` field — `FIXED`
146+
`runner/dispatcher.py:_score_c1_grounding` now passes the output's
147+
`limitations` list to `score_facts` as the documented fallback search target.
148+
Covered by `test_c1_grounding_searches_limitations`.
149+
150+
### M-2. Half the judge spend is wasted — `FIXED`
151+
`score_rubric` now accepts a `dimensions` subset (full-rubric weight
152+
validation unchanged), and the dispatcher judges only the three C2-owned
153+
dimensions (`insight_quality`, `evidence_linkage`, `structure_usability`).
154+
This halves judge calls: 3 per run instead of 6. Covered by
155+
`test_dimensions_subset_judges_only_those` and
156+
`test_c2_judge_not_called_for_non_c2_dimensions`.
157+
158+
### M-3. Small task count → noisy rankings — `DISCLOSED`
159+
26 tasks total (11 operations, 5 outcomes, 10 equity_research); per-track N
160+
is as low as 5. Disclosure added to `docs/methodology.md` ("Statistical
161+
precision") and to the grade-app public methodology page (treat close calls
162+
as ties). Bootstrap CIs over per-task composites remain a nice-to-have.
163+
164+
### M-4. Public fixtures start the contamination clock — `DISCLOSED`
165+
Disclosure + policy added to `docs/methodology.md` ("Benchmark
166+
contamination"): fixtures are regenerated (new seed + recomputed
167+
`ground_truth.json`) per numbered benchmark version, and scores are only
168+
comparable within a version. Plain-language version added to the grade-app
169+
methodology page.
170+
171+
### M-5. Judge cannot verify evidence against data — `DISCLOSED`
172+
Disclosure added to `docs/methodology.md` ("Judge scope") and, in plain
173+
language, to the grade-app methodology page: the judge scores against gold
174+
material; numbers are verified separately by the deterministic C1 scorer.
175+
176+
### M-6. Bare-float judging (no rationale) — `FIXED`
177+
The judge prompt now asks for a brief (2-3 sentence) justification followed
178+
by a final `SCORE: <float>` line; parsing prefers the last `SCORE:` line and
179+
falls back to the last float, so numbers quoted in the justification are
180+
never mistaken for the verdict. Non-reasoning judge `max_tokens` raised
181+
16 → 384 to fit the rationale. Locked in before the first official run, so
182+
no published scores change.
183+
184+
---
185+
186+
## Launch model lineup (verified against OpenRouter 2026-06-10)
187+
188+
| Leaderboard entry | OpenRouter slug | Pricing ($/M in / out) |
189+
|---|---|---|
190+
| Claude Opus 4.8 | `anthropic/claude-opus-4.8` | (see openrouter.ai) |
191+
| Claude Sonnet 4.6 | `anthropic/claude-sonnet-4.6` | |
192+
| Claude Haiku 4.5 | `anthropic/claude-haiku-4.5` | |
193+
| GPT-5.5 (xhigh) | `openai/gpt-5.5` + `reasoning.effort=xhigh` | 5.00 / 30.00 |
194+
| GPT-5.5 (high) | `openai/gpt-5.5` + `reasoning.effort=high` | 5.00 / 30.00 |
195+
| GPT-5.5 (medium) | `openai/gpt-5.5` + `reasoning.effort=medium` | 5.00 / 30.00 |
196+
| GPT-5.5 (low) | `openai/gpt-5.5` + `reasoning.effort=low` | 5.00 / 30.00 |
197+
| Gemini 3.5 Flash | `google/gemini-3.5-flash` | 1.50 / 9.00 |
198+
| Gemini 3.1 Pro | `google/gemini-3.1-pro-preview` | 2.00 / 12.00 |
199+
| Nemotron 3 Ultra | `nvidia/nemotron-3-ultra-550b-a55b` | 0.50 / 2.50 |
200+
| GPT-OSS 120B | `openai/gpt-oss-120b` | ~0.10 / 0.45 |
201+
202+
Notes:
203+
- Gemini 3.1 Pro only exists as a `-preview` slug; re-verify before the run.
204+
- The four GPT-5.5 effort rows depend on H-5 (reasoning passthrough).
205+
- Open-weight rows (Nemotron, GPT-OSS) chosen for brand recognition with US
206+
educators; Llama and Mistral dropped per product decision 2026-06-10.
207+
208+
## Run protocol for the official leaderboard (after fixes)
209+
210+
```bash
211+
python -m scripts.run_models \
212+
--models <comma-separated slugs above> \
213+
--pack operations --runs 5 --judge --out results/operations
214+
# repeat for outcomes and equity_research, then merge per-pack results
215+
```
216+
217+
- `--runs 5` (never 1 — see H-3), temperature 1.0 (default), judge required.
218+
- Judge is auto-selected per candidate (Opus 4.8 default; GPT-5.5 xhigh for
219+
all Claude-family candidates — no judge shares a family with its
220+
candidate). `--judge-model` overrides.
221+
- Record judge model + version in the methodology page alongside results,
222+
including the per-candidate judge assignment and the calibration caveat
223+
noted under C-3.

0 commit comments

Comments
 (0)