Why
Today, when something goes wrong (missing jq, wrong Python, opencode not on PATH, API key not set, registry not initialized), the user gets a cryptic error somewhere downstream. A doctor subcommand that prints a green/red checklist of every prerequisite would cut first-run friction enormously.
What to build
A new scripts/eval/doctor.sh invoked as eval-harness doctor that checks and prints:
eval-harness doctor
[✓] bash 5.2.15
[✓] jq 1.7.1
[✓] python3 3.11.4 + yaml module
[✓] opencode 1.15.2 on PATH
[✓] ANTHROPIC_API_KEY set (sk-ant-...e4f2)
[✗] OPENCODE_SKILLS_ROOT not set — will walk up from cwd
[✓] registry initialized at /Users/x/.eval-harness/state/registry.json
[✓] this repo is enabled in registry
[✓] pricing.json fresh (12 days old, max 60)
Status: 8/9 OK. See https://github.com/nano-step/eval-harness/blob/main/docs/concepts.md for context on warnings.
Exit codes:
0 if all checks pass (warnings allowed)
64 if any check fails (missing required tool)
Acceptance criteria
Good first issue because
- Self-contained new file
- No coupling to score / attribute / stability
- Pattern mirrors existing
preflight.sh which can be cannibalized for check helpers
- 100-150 lines of bash
Comment to claim.
Why
Today, when something goes wrong (missing jq, wrong Python, opencode not on PATH, API key not set, registry not initialized), the user gets a cryptic error somewhere downstream. A
doctorsubcommand that prints a green/red checklist of every prerequisite would cut first-run friction enormously.What to build
A new
scripts/eval/doctor.shinvoked aseval-harness doctorthat checks and prints:Exit codes:
0if all checks pass (warnings allowed)64if any check fails (missing required tool)Acceptance criteria
scripts/eval/doctor.sh(set -euo pipefail)run.shvia adoctorsubcommandscripts/eval/tests/doctor.shcovering: all pass, one fail, one warningeval-harness doctor" lineGood first issue because
preflight.shwhich can be cannibalized for check helpersComment to claim.