B1 — Grounded Retrieval tasks - #10
Merged
Merged
Conversation
Add 6 Track 1 tasks (tasks.jsonl) covering direct counts, attendance-rate computation, date-range filtering, relative-time interpretation, and IEP/non-IEP scope comparison against the Operations fixture pack. Include a verification test suite (37 tests) that schema-validates every task and independently recomputes every numeric gold fact from the fixture CSVs, confirming correctness before merge.
|
…o_show) definition The October cancellation gold fact used a cancelled-only definition (21.1%), contradicting the canonical monthly_cancellation_rate in ground_truth.json / generator.py (cancelled + no_show = 27.9%, also used by Track 2). Reconcile the task's gold facts, insights, limitations, and reference outline to the composite definition, and update the verification test to recompute with no_show included.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
benchmark/tracks/grounded_retrieval/tasks.jsonlagainst the Operations fixture pack, covering: direct counts (students, groups, schools), program-wide attendance-rate computation, monthly rate comparison (Sep/Oct/Nov), October cancellation-spike identification, IEP vs non-IEP attendance gap, and November relative-time retrieval.benchmark/tracks/grounded_retrieval/README.mddescribing track scope, fixture files, and authoring conventions.tests/unit/test_track_grounded_retrieval.pywith 37 tests: 6 parametrized schema-validity tests plus per-task gold-fact recomputation tests that independently derive every numeric value from the fixture CSVs.Test plan
pytest tests/unit/test_track_grounded_retrieval.py— 37 tests, all passpytest(full suite) — 141 tests, all passruff check .— no issuesruff format --check .— no changes neededmypy benchmark runner— no issuespre-commit run --all-files— all hooks passImplements platform-ai#589
🤖 Generated with Claude Code