Evals: vally 0.8.0 skills migration + comparative baselines + docs#35
Open
IEvangelist wants to merge 6 commits into
Open
Evals: vally 0.8.0 skills migration + comparative baselines + docs#35IEvangelist wants to merge 6 commits into
IEvangelist wants to merge 6 commits into
Conversation
vally 0.6.0 changed the default to "no skills"; skills are now declared per-eval via environment.skills. The CI ci-gate (which passes no --skill-dir) was therefore silently running every stimulus with no skills loaded - measuring the baseline instead of gating the skills. - Declare eval-level environment.skills (the skill under test plus its in-repo dependency closure) in all 6 skills/*/evals/eval.yaml, restoring the real ci-gate under 0.6.0. - Add skill-lift.experiment.yaml: a comparative baseline that runs every eval twice along /environment/skills (with-skills treatment vs no-skills baseline) so each skill's measured lift can be reported. - Add .github/workflows/skill-experiment.yml: weekly + dispatch, informational (never a gate), token-preflight soft-skip, summarizes lift to the step summary and uploads the skill-lift-results artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- README.md: document environment.skills (the 0.6.0 "no skills by default" change), the hybrid skill-loading convention, expect_skills/reject_skills, and a new "Skills & baselines" section covering skill-lift.experiment.yaml and `vally experiment run`. Fix the --skill-dir flag description, the `vally compare --run-a/--run-b` syntax, the per-skill run command (no more --skill-dir), and add the skill-experiment.yml workflow to the CI table. - AUTHORING.md: rewrite off the obsolete tasks/<id>.yaml + --context-dir + trigger_tests.yaml format onto the canonical inline-stimuli EvalSchema. Correct the grader-type table (text/code -> prompt, output-contains, output-not-contains, output-matches, skill-invocation, pairwise/panel), replace the trigger-tests section with skill-invocation routing guidance, and refresh the checklist and pitfalls for the 0.6.0 skills model. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the repo’s vally eval specs to the 0.6.0 “no skills by default” behavior by explicitly declaring environment.skills, and adds an experiment + workflow to measure comparative “skill lift” against a no-skills baseline. It also updates eval documentation to reflect the new schema and conventions.
Changes:
- Add
environment.skillsto each skill’sskills/*/evals/eval.yamlsoci-gateruns with the intended skills under vally 0.6.0. - Introduce
skill-lift.experiment.yamlplus a scheduled/dispatch workflow to run with-skills vs no-skills baselines and publish results. - Update
evals/README.mdandevals/AUTHORING.mdto document the new schema and baseline/experiment patterns.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/aspireify/evals/eval.yaml | Declares eval-level environment.skills for aspireify (+ dependency). |
| skills/aspire/evals/eval.yaml | Declares eval-level environment.skills for routing tests (full skill set). |
| skills/aspire-orchestration/evals/eval.yaml | Declares eval-level environment.skills for orchestration evals. |
| skills/aspire-monitoring/evals/eval.yaml | Declares eval-level environment.skills for monitoring evals. |
| skills/aspire-init/evals/eval.yaml | Declares eval-level environment.skills including transitive in-repo dependencies. |
| skills/aspire-deployment/evals/eval.yaml | Declares eval-level environment.skills for deployment evals. |
| skill-lift.experiment.yaml | Adds a two-variant experiment that varies /environment/skills (treatment vs baseline). |
| evals/README.md | Documents vally 0.6.0 skills loading, hybrid convention, and the lift experiment/workflow. |
| evals/AUTHORING.md | Updates authoring guidance to inline-stimuli EvalSchema and routing assertions. |
| .github/workflows/skill-experiment.yml | Adds an informational workflow to run the lift experiment and upload results. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+16
to
+18
| # vally 0.6.0 loads no skills unless declared here (paths resolve relative to | ||
| # this eval file). Load aspireify plus its in-repo dependency: aspireify | ||
| # validates wiring by running `aspire start` (aspire-orchestration territory). |
Comment on lines
+49
to
+51
| echo "**To enable it**, a maintainer needs to add the secret once —" | ||
| echo "see [\`evals/README.md\` → CI authentication](../blob/main/evals/README.md#ci-authentication)." | ||
| } >> "$GITHUB_STEP_SUMMARY" |
Comment on lines
+91
to
+100
| { | ||
| echo "## 📊 Skill-lift baseline" | ||
| echo "" | ||
| echo "Per-variant pass summary (\`with-skills\` treatment vs \`no-skills\` baseline):" | ||
| echo "" | ||
| echo '```' | ||
| # The per-variant "✔/✘ <variant>: N/M evals passed" lines. | ||
| grep -E 'evals passed|Run ID' experiment-output.txt || cat experiment-output.txt | ||
| echo '```' | ||
| echo "" |
The Skill Eval PR gate failed after ~78 minutes because none of the eval specs declared a scoring config. Under vally 0.6.0 a missing `scoring` block defaults to the strictest binary verdict: every grader must pass on every trial, so a single grader miss (or one slow trial) fails the whole stimulus and reds the gate — even when the agent did the work correctly. Make the gate meaningful and reliable: * Add `scoring.threshold: 0.7` to all six eval specs. Each stimulus is now scored by the share of graders that pass, and the eval gates on the aggregate. Genuine regressions still drag the aggregate below the bar; nondeterminism and harness artefacts (e.g. graders scanning the chat message for code the agent wrote into edited files) no longer hard-fail an otherwise-correct run. * Bump the per-trial timeout 120s -> 180s so routing/capability trials that legitimately need a little longer stop erroring out and scoring 0. * Router eval: convert the ten `should_trigger_*` graders from `skill-invocation required:[aspire]` to outcome-based `prompt` graders. Under the copilot-sdk executor the agent satisfies these prompts by running the aspire CLI directly (a `bash` tool call) or by going straight to the matching sub-skill, which is correct behavior but emits no top-level `skill_activation` event — so the old graders produced false negatives even when the task was done perfectly (e.g. it ran `aspire init`, pulled `aspire logs`/OTEL, recognized `apphost.ts`). The new graders assert the correct Aspire workflow was engaged and still catch real routing regressions. The complementary should_not_trigger stimuli keep their skill-invocation `disallowed` guard against over-triggering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous gate run (28388912985) scored every eval above the 0.7 threshold (init 86.6%, monitoring 80.3%, orchestration 77.7%) yet still failed, because vally exits 1 on any execution/tooling error regardless of score. Three trials errored with "Timeout after 300000ms waiting for session.idle": - orchestration should_not_trigger_13 "Run aspire dashboard run standalone" - monitoring should_trigger_08 (also aspire dashboard run) [pre-empted] - aspireify should_not_trigger_07 "Open the Aspire dashboard" [pre-empted] - init should_trigger_01 (agent ran a slow `aspire new` restore) - monitoring should_not_trigger_04 (transient recursive grep) Root cause: `aspire dashboard run` launches the standalone dashboard as a foreground daemon — even with no AppHost present — so the agent turn never returns and the session.idle wait times out. A single hung trial hard-fails the whole eval, and no timeout/retry can save an inherently-blocking command. These are pure routing stimuli (skill-invocation graders only assert which skill activates), so phrasing them as questions tests routing identically while letting the agent answer instead of launching a daemon or running a long template restore. All three `aspire dashboard run` prompts and the two observed transient hangs are reworded; graders are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
vally 0.8.0 (published 2026-07-08) is the current release; CI installs it as the npm 'latest'. The eval schema our specs rely on (defaults, type: capability, environment.skills, graders, scoring) is unchanged from 0.6.0, so this is a version-reference refresh plus a couple of accuracy fixes: - README.md / AUTHORING.md: retag 'vally 0.6.0' notes and the Skills & baselines anchor to 0.8.0; correct 'config.model'/'config.runs' to 'defaults.*' and move expect_skills/reject_skills docs under 'constraints' (their 0.8.0 home). - eval.yaml comments: 'vally 0.6.0 loads no skills' -> '0.8.0'. Verified locally with vally 0.7.0: 'vally lint skills' 6/6, per-spec eval-spec lint all valid, and 'vally experiment run --dry-run' plans cleanly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788994fa-ffbf-4722-a7b8-0dcc3b174dbf
IEvangelist
changed the title
runs >= 2 disables vally's timeout/rate-limit retries, so a single trial that wanders past the per-trial budget errors and hard-fails the whole eval (evalHadExecutionErrors) even when the aggregate score clears the threshold. Doubling the wall-clock budget to 600s gives slow trials room to finish instead of erroring, removing the pre-existing CI flakiness. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788994fa-ffbf-4722-a7b8-0dcc3b174dbf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stands up the evals-driven improvement loop on vally 0.8.0 (the current release; CI installs it as the npm
latest) and adds the comparative baselines (same stimuli, no skills) needed to measure each skill's lift.Why
vally 0.6.0 flipped the default to no skills — skills are declared per-eval via
environment.skills— and that behavior still holds in 0.8.0. Because the CIci-gate(vally eval --suite ci-gate) passes no--skill-dir, every stimulus had silently started running with no skills loaded — i.e. it was already measuring the baseline instead of gating the skills. This PR restores the real gate and makes the baseline a first-class, reportable artifact.Changes
skills/*/evals/eval.yamlnow declares a top-levelenvironment.skillswith the skill under test plus its in-repo dependency closure (hybrid loading: capability specs load the closure; the router +area: routingstimuli load the full set). This is the critical fix that makesci-gatehonest again.skill-lift.experiment.yaml(repo root) — avally experimentthat runs every spec twice along/environment/skills(with-skillstreatment vsno-skillsbaseline). The pass-rate delta is each skill's measured lift..github/workflows/skill-experiment.yml— weekly +workflow_dispatch, informational (never a gate). The no-skills baseline is expected to fail grading, so the experiment runs tolerantly, summarizes the lift to the step summary, and uploads theskill-lift-resultsartifact. Mirrors the existing token-preflight soft-skip pattern.evals/README.mdgains a "Skills & baselines (vally 0.8.0)" section (environment.skills, hybrid convention,constraints.expect_skills/reject_skills, experiments).evals/AUTHORING.mdis migrated onto the canonical inline-stimuliEvalSchema, with a corrected grader-type table andskill-invocationrouting guidance.vally 0.8.0 refresh
This PR was originally authored against vally 0.6.0. The eval schema the specs rely on (
defaults,type: capability,environment.skills, graders,scoring) is unchanged through 0.8.0, so the migration is a version-reference refresh plus two accuracy fixes:config.model/config.runs→defaults.*, andexpect_skills/reject_skillsdocumented underconstraints(their 0.8.0 home). Nopanelgraders are used, so 0.8.0's strict-judge change doesn't apply, and eval names don't collide with reserved run-output artifact names.Validation
vally lint skills→ 6/6 pass; per-specvally lint --eval-spec→ all valid.vally experiment run skill-lift.experiment.yaml --dry-run→ 12 plans (6 evals × 2 variants); eachno-skillsplan shares one bare config hash and eachwith-skillsplan has a distinct hash → confirms the baseline strips skills and the treatment inherits each spec's closure.ci-gateeval.Notes
COPILOT_GITHUB_TOKENrepo secret (seeevals/README.md → CI authentication). Without it they soft-skip green.awaitnoise in TypeScript AppHost examples #31, Bicep Converter to AppHost #29) follow in a stacked PR on top of this branch.