Skip to content

Add eval-harness doctor — preflight diagnostics command #32

Description

@hoainho

Why

Today, when something goes wrong (missing jq, wrong Python, opencode not on PATH, API key not set, registry not initialized), the user gets a cryptic error somewhere downstream. A doctor subcommand that prints a green/red checklist of every prerequisite would cut first-run friction enormously.

What to build

A new scripts/eval/doctor.sh invoked as eval-harness doctor that checks and prints:

eval-harness doctor

[✓] bash 5.2.15
[✓] jq 1.7.1
[✓] python3 3.11.4 + yaml module
[✓] opencode 1.15.2 on PATH
[✓] ANTHROPIC_API_KEY set (sk-ant-...e4f2)
[✗] OPENCODE_SKILLS_ROOT not set — will walk up from cwd
[✓] registry initialized at /Users/x/.eval-harness/state/registry.json
[✓] this repo is enabled in registry
[✓] pricing.json fresh (12 days old, max 60)

Status: 8/9 OK. See https://github.com/nano-step/eval-harness/blob/main/docs/concepts.md for context on warnings.

Exit codes:

  • 0 if all checks pass (warnings allowed)
  • 64 if any check fails (missing required tool)

Acceptance criteria

  • New scripts/eval/doctor.sh (set -euo pipefail)
  • Wired into run.sh via a doctor subcommand
  • Each check is its own function so they're easy to extend
  • Test at scripts/eval/tests/doctor.sh covering: all pass, one fail, one warning
  • README quickstart section gets a "When in doubt, run eval-harness doctor" line

Good first issue because

  • Self-contained new file
  • No coupling to score / attribute / stability
  • Pattern mirrors existing preflight.sh which can be cannibalized for check helpers
  • 100-150 lines of bash

Comment to claim.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions