Search results

BigBench accuracy always 0% with reasoning models: grader scores reasoning+content instead of final answer

Status: Open.

#1136 In ai-dynamo/aiperf;

· Maze999 opened on Jul 9, 2026

--extra-inputs payload_template is dropped when a --config file is supplied

Status: Open.

#1135 In ai-dynamo/aiperf;

· Lokiiiiii opened on Jul 9, 2026

Fix: CLI flag binding to enable export_outputs_json

Status: Open.

#1133 In ai-dynamo/aiperf;

· Lokiiiiii opened on Jul 8, 2026

Support per-URL custom headers for repeated --url targets

Status: Open.

#1125 In ai-dynamo/aiperf;

· ajcasagrande opened on Jul 7, 2026

Weka loader caps before context filtering and silently wraps a too-small conversation pool

AgentXFeature for AgentXFeature for AgentX

bugSomething isn't workingSomething isn't working

Status: Open.

#1106 In ai-dynamo/aiperf;

· PeaBrane opened on Jul 3, 2026

Template model fails to parse top level JSON Array in Response

Status: Open.

#1090 In ai-dynamo/aiperf;

· Lokiiiiii opened on Jul 2, 2026

[Contribution] Feature: Persistent system prompt for multi-turn datasets

Status: Open.

#1062 In ai-dynamo/aiperf;

· hatemfaheem opened on Jun 23, 2026

request_content_type=multipart/form-data should be supported for ASR models like standlalone whisper

Status: Open.

Feature

#1059 In ai-dynamo/aiperf;

· Maze999 opened on Jun 17, 2026

--tokenizer-trust-remote-code cannot load DeepSeek-V3.2-Exp tokenizer (model_type=deepseek_v32) — no config fallback for an unregistered model_type without auto_map

Status: Open.

#1047 In ai-dynamo/aiperf;

· janbernloehr opened on Jun 10, 2026

credit pipeline: surface mid-run issuance-task exceptions instead of swallowing them (follow-up to #1039)

Status: Open.

#1041 In ai-dynamo/aiperf;

· ajcasagrande opened on Jun 6, 2026

Add support for agentic (multi-step, stateful, tool-calling) evals

Status: Open.

#1038 In ai-dynamo/aiperf;

· abatilo opened on Jun 5, 2026

Add swe_rebench_openhands public dataset loader for agentic multi-turn benchmarking

Status: Open.

#1019 In ai-dynamo/aiperf;

· Lokiiiiii opened on Jun 2, 2026