Skip to content

[Fix] fix MathCanvas-Bench md5 & summarize_mathcanvas_results#1502

Open
shiwk24 wants to merge 1 commit intoopen-compass:mainfrom
shiwk24:fix/mathcanvas-bench-eval
Open

[Fix] fix MathCanvas-Bench md5 & summarize_mathcanvas_results#1502
shiwk24 wants to merge 1 commit intoopen-compass:mainfrom
shiwk24:fix/mathcanvas-bench-eval

Conversation

@shiwk24
Copy link
Copy Markdown
Contributor

@shiwk24 shiwk24 commented Mar 31, 2026

Summary

Fixes two issues affecting MathCanvas-Bench usage and reporting.

Changes

  • Dataset MD5: Update MathCanvas-Bench checksum in image_vqa.py so the local TSV matches the current file and avoids unnecessary re-download / mismatch warnings.
  • Summary stats (summarize_mathcanvas_results in mathcanvas.py): Align by_question_image_count and by_knowledge keys with the dataset so per-category and image-presence metrics count correctly (previously keys did not match).

Notes

No change to the judging logic; reporting / aggregation only, plus the hash update.

@clintonsteiner
Copy link
Copy Markdown

@dependabot rebase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants