Skip to content

evals: single canonical v4 task root + dataset file overrides#2405

Draft
shriyatheunicorn wants to merge 1 commit into
evals-v4-portfrom
shriyalolabattu/stg-2671-run-selection
Draft

evals: single canonical v4 task root + dataset file overrides#2405
shriyatheunicorn wants to merge 1 commit into
evals-v4-portfrom
shriyalolabattu/stg-2671-run-selection

Conversation

@shriyatheunicorn

@shriyatheunicorn shriyatheunicorn commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Part of STG-2671.

Single canonical v4 task root

tasks/bench-v4 is the one v4 discovery root — the migration-window copy inside the v4-spike checkout was verified to be a strict subset (60/60 tasks semantically identical) and deleted. The name-collision dedupe guard stays, so multi-root tiers can never double-count (previously run act --sdk v4 ran 64 cases instead of 40).

Dataset file overrides

EVAL_WEBVOYAGER_FILE / EVAL_WEBTAILBENCH_FILE (mirroring EVAL_GAIA_FILE) point runs at curated slices — the stock files are site/category-ordered, so -l N alone only ever reaches the first site's rows (every WebVoyager smoke so far was Allrecipes-only for exactly this reason).


Summary by cubic

Set tasks/bench-v4 as the single v4 discovery root and added dataset file overrides for WebVoyager and WebTailBench. This aligns v4 evals with STG-2671 and lets short runs target curated slices for better coverage.

  • New Features

    • Discover v4 bench tasks only from tasks/bench-v4 (v4-spike copy removed).
    • Support EVAL_WEBVOYAGER_FILE and EVAL_WEBTAILBENCH_FILE to load curated JSONL slices; defaults unchanged.
  • Bug Fixes

    • Deduplicate tasks by name across multi-root tiers so each case runs once.

Written for commit d5dead1. Summary will update on new commits.

Review in cubic

- discovery: tasks/bench-v4 is the single v4 root (the migration-window
  copy inside the v4-spike checkout was verified to be a strict subset
  and deleted); name-collision dedupe guard stays so multi-root tiers
  can never double-count a task.
- suites: EVAL_WEBVOYAGER_FILE / EVAL_WEBTAILBENCH_FILE overrides
  (mirroring EVAL_GAIA_FILE) so runs can target curated slices - the
  stock files are site/category-ordered, so `-l N` alone only ever
  reaches the first site's rows.

Part of STG-2671
@changeset-bot

changeset-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d5dead1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant