feat(eval): add GAIA benchmark for native rllm eval

[Zayed024]

Summary

Adds GAIA as a native benchmark so rllm eval gaia runs it. GAIA (General AI
Assistants) is a headline long-horizon agentic eval reported across current
frontier model releases. Integration follows the existing rows + transform +
reward_fn pattern (cf. hotpotqa/hle) — no sandbox/Docker. Graded by GAIA's
official quasi-exact-match scorer, ported verbatim for leaderboard-comparable
numbers.

Type of change

• Feature
• Test

What changed

• rllm/eval/reward_fns/gaia.py: gaia_reward_fn (evaluate) + the official
  question_scorer (number / list / string normalization), attributed to the
  GAIA scorer (Apache-2.0).
• rllm/data/transforms.py: gaia_transform — maps GAIA rows to standard QA
  format; skips file-attachment tasks (text-only subset for now).
• rllm/eval/evaluator_loader.py: register gaia_reward_fn.
• rllm/registry/datasets.json: gaia entry (source gaia-benchmark/GAIA,
  validation split, gated: true, default_agent: search).
• tests/eval/test_gaia_reward.py: 17 tests — scorer (number/list/string edge
  cases), transform filtering, evaluate via load_evaluator, registry wiring.

Validation

• pytest tests/eval/test_gaia_reward.py -q → 17 passed
• ruff check + ruff format --check clean (pinned v0.11.4)
• Full rllm eval gaia run pending (GAIA is HF-gated — needs HF_TOKEN —
  and a search-capable agent; evaluator/transform unit-tested in isolation)

Open questions for reviewers

• Scope: text-only subset first (file-attachment tasks skipped) — want file-tool
  support as a follow-up, or in this PR?
• default_agent: search — is that the right harness for GAIA, or prefer react?
• τ²-bench is the natural next agentic integration — is it on the roadmap / in flight?

Breaking changes / migration notes

• None

Related issues / PRs

• None

[jeffreysijuntan]

Might need some investigation what harness frontier model use to evaluate on GAIA, and then implement that in rLLM. Can you past screenshots of results from rllm eval gaia to confirm the whole thing is working E2E?

[Zayed024]

sure i will

…

On Fri, 26 Jun 2026 at 23:27, Sijun ***@***.***> wrote: *jeffreysijuntan* left a comment (rllm-org/rllm#703) <#703 (comment)> Might need some investigation what harness frontier model use to evaluate on GAIA, and then implement that in rLLM. Can you past screenshots of results from rllm eval gaia to confirm the whole thing is working E2E? — Reply to this email directly, view it on GitHub <#703?email_source=notifications&email_token=BCKA7TJIJQCSYRVERSUV7FL5B22RLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBRGIYDMOBUG432M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4812068477>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCKA7TKHQJDYTRMTU4XLYUD5B22RLAVCNFSNUABFKJSXA33TNF2G64TZHM4TEMRUGA3DKMRSHNEXG43VMU5TINZVGEZTKOBQG432C5QC> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Zayed024]

E2E confirmed: rllm eval gaia with gpt-4o → 25% (3/12), 0 errors, agent does search→extract→answer. Fixed a
bug (one Step per LLM turn) that tripped the trace enricher on parallel tool calls.