v0.4.2 — Hardening release (closes all 8 audit BLOCKERs)

v0.4.2: hardening release — close all 8 BLOCKERs from 2026-05-30 audits

v0.4.1 — fix npm-link symlink resolution

[0.4.1] — 2026-05-30

Fixed

• Install via npm link / npm install -g was broken. Top-level entrypoints (run.sh, twotier.sh, accept.sh, baseline.sh, install-hooks.sh) resolved BASH_SOURCE[0] via $(dirname …) only, which returned the symlink directory (the npm bin/), not the package's actual scripts/eval/ directory. Each entrypoint now walks the symlink chain portably.

v0.4.0 — heuristic auto-fix proposer

[0.4.0] — 2026-05-29

Added

• Heuristic auto-fix proposer — scripts/eval/lib/autofix.sh attaches .fix_proposal to every FAILED check whose failure mode is mechanically diagnosable: output_contains, output_not_contains, jq_path_contains, file_exists, shell (exact / min / regex). Each proposal carries kind, confidence, instruction, patch_snippet, and auto_apply: false. llm_judge and unknown kinds yield fix_proposal: null — the harness won't guess prose. Gated by EVAL_AUTOFIX (default 1). (PR #13)

Verified

• 11 test suites green (v0.3.0's 10 + autofix).

v0.3.0 — LLM judge + pr-code-reviewer demo + 2-tier mode

[0.3.0] — 2026-05-29

Added

• LLM judge — new check kind llm_judge for prose-output skills. Calls Anthropic Messages API directly via curl with ANTHROPIC_API_KEY. Default claude-sonnet-4-6; configurable to claude-opus-4-7 via EVAL_LLM_JUDGE_MODEL or per-check judge_model. 3-sample majority voting (samples: N per check). Returns verdict: null with explicit reason when API key missing, curl fails, response unparseable, or majority null — never fabricates a verdict. (PR #9)
• pr-code-reviewer demo skill — second skill alongside omo-session-distiller. 3 prose cases exercising the llm_judge check kind: adversarial SQL injection (must flag), false-positive control trivial rename (must approve), and partial-failure billing risk (must flag). (PR #10)
• 2-tier mode — --mode={smoke|full|2tier}. smoke (cheap haiku + 1 sample) is default. full (sonnet-4-6 + 3 samples) is the canonical pass. 2tier runs smoke first, then re-runs only the FAILED cases with full. Trigger string 2tier-escalation for history transparency. (PR #11)

Fixed

• run.sh SCRIPT_DIR shadowing — lib/diff.sh reassigned the variable on source, breaking sibling-script resolution from run.sh. Now uses RUN_SCRIPT_DIR for run.sh-owned paths.

Verified

• 10 test suites green: all of v0.2.0's 8 + llm_judge_unit + twotier_mode.

v0.2.0 — project config, model override, lockfile, registry, pricing, stability, stop-hook scaffold

[0.2.0] — 2026-05-29

Added

• Per-case model override — case YAML .model field. Resolution: case YAML > EVAL_MODEL > OPENCODE_MODEL > built-in default. Recorded in env-manifest so MODEL_CHANGED attribution fires correctly. (PR #1)
• Project-config layer — .opencode/eval-harness.yaml walked up from cwd. Settings: model, budget_usd, max_seconds, skills_root, llm_judge.model. Env vars win when explicitly set. (PR #2)
• opencode Stop hook scaffold — scripts/eval/hooks/opencode-stop.sh parses OPENCODE_CHANGED_FILES and re-runs evals for touched skills. Gated on opencode >= 1.16 (no-op until plugin API stabilizes). (PR #3)
• Per-repo opt-in registry — scripts/eval/lib/registry.sh manages ~/.config/opencode/eval-harness/registry.yaml. Automated triggers (pre-push, sync-publish, stop-hook) skip when current repo isn't enabled. Required for 43-repo workspace support. (PR #4)
• Per-(case,trigger) lockfile coordination — flock(1) wraps the manifest+spawn+score critical section; mkdir-atomic fallback for flock-less platforms (macOS). EVAL_LOCK_TIMEOUT env (default 300s). (PR #5)
• pricing.json + dollar cost reporting — curated rates for haiku-3-5, sonnet-4-6, opus-4-7. Per-case cost.usd in results.json. Total at summary.total_cost_usd. Staleness warning (default 60 days); EVAL_FAIL_ON_STALE_PRICING=1 to gate. (PR #6)
• 3-sample stability on critical path — --stability-samples=N flag + EVAL_STABILITY_SAMPLES env. On FAIL, runs N-1 more samples. Records byte-identicity. Tags attribution flaky:true when samples diverge. (PR #7)

Changed

• lib/yq-shim.sh (_yq.py) now handles -o=json argv form, []? iteration suffix, and // [] fallback for empty lists.
• lib/diff.sh results schema gains .cases[i].cost, .cases[i].stability, .summary.total_cost_usd.

Verified

• 8 test suites all green: regression_inject, case_model_override, project_config, registry, lock_concurrency, pricing, stability_inline, stop_hook.
• v0.1.0 wire-format remains backward compatible (new fields are additive).

v0.1.1 — patch: model ID + demo path + factors README + SQS-1 honesty

Patch release bundling four follow-up commits since v0.1.0.

Fixes

• spawn.sh: default model ID changed to anthropic/claude-3-5-haiku-latest. v0.1.0's claude-haiku-3-5 was rejected by opencode 1.15.10 with "Model not found." Every real run now resolves. (commit 144c8e1)
• tests/regression_inject.sh: demo skill path resolves from multiple candidate roots, so npm test works after npm install -g (not just inside a repo clone). (commit dffdc99)

Documentation

• README: new sections — "What this harness scores" (5 check kinds + 4 attribution fields), "The review workflow" (Mermaid flowcharts for pre-push + sync-publish gates), "How to verify the harness is actually running these factors." (commit 2e8de78)
• standards/skill-quality-v1.md: SQS-1 re-framed as draft heuristic with honest 13🟢 / 7🟡 / 10🔴 tier split per check. (commit 8c6412c)

No API changes

run, baseline, accept, status, promote, trend all unchanged. Same case YAML schema. Same 6-field FAIL output. Same exit codes (0 / 12 / 13). Pure patch release.

Upgrade

npm install -g @nano-step/eval-harness@latest

See CHANGELOG.md for full notes.

v0.1.0 — initial release

Initial release

Behavior-regression eval harness for opencode skills.

Scope (v0.1.0)

• Structured-output skills only. Prose-output skills (pr-code-reviewer, od-workflow, blog-workflow, idea-workflow) deferred to v0.3 (LLM judge).
• Deterministic mode (T=0, k=1). pass@k deferred to v0.2.
• 3 triggers: manual via eval-harness run, git pre-push, sync-skill-to-manager pre-publish (opt-in per skill).
• opencode 1.15.10 verified. Earlier/later versions: file an issue.

Architecture

• Bash + jq + python3 (yq-shim for YAML). No daemon, no Unix socket, no Node CLI binary.
• 4-class attribution: SKILL_CHANGED / FIXTURE_STALE / MODEL_CHANGED / UNKNOWN_DRIFT.
• 6-field FAIL schema: failed_check_id, expected, actual, diff_hint, transcript_span (nullable), env_delta.
• One-command rerun in every FAIL output.
• Warn-only by default for 7 days. eval-harness promote enables blocking.

Safety rails

• Two-stage accept: default keeps env_manifest; --bless-env requires confirmation (avoids silent model drift).
• EVAL_BYPASS=1 escape hatch with audit log to history.ndjson.
• Ephemeral sandbox per case (fresh HOME, OPENCODE_CONFIG_DIR, NANO_BRAIN_ROOT, cwd).

Design provenance

Designed via the deep-design multi-agent pipeline:

• Phase 1: Metis (scope/risk) + Oracle (architecture) in parallel
• Phase 1.5: cross-critique (Metis ↔ Oracle)
• Phase 2: confidence-scored synthesis → 18 Settled Decisions
• 15 of 18 fully implemented; 3 partial implementations documented in README.

Demo

npm test

Runs scripts/eval/tests/regression_inject.sh end-to-end. All 5 acceptance assertions green:

• verdict=REGRESSION
• attribution=SKILL_CHANGED
• regression list contains atom-tags-decision-architecture
• failed_check_id populated
• warn-only exit 0 (not yet promoted)

Known limitations

• 3-sample stability check implemented but not wired into critical path (deferred to v0.2)
• pricing.json dollar conversion + staleness gate deferred to v0.2
• Per-(case,trigger) lockfile deferred to v0.2 (only needed when Stop hook lands)

Roadmap

• v0.2.0: opencode Stop hook + per-repo opt-in registry + lockfile coordination
• v0.3.0: LLM judge (claude-haiku, cross-model debias, 3-sample majority) + pr-code-reviewer demo
• v0.4.0: heuristic auto-fix proposer (constrained to literal/regex checks)

Roadmap-adjacent: skill-reviewer

Separate skill (not in this repo) consuming standards/skill-quality-v1.md for design review (Job 3). Eval-harness measures BEHAVIOR REGRESSION; skill-reviewer measures DESIGN QUALITY.

See README.md for full installation + usage.