Several retail tasks reward success even when the listed calculate(...) action is absent

[gss10282023]

I think several retail tasks may have an inconsistency between listed calculate(...) actions and what the released reward actually requires for success.

In the cases below, the official result is reward=1.0 with reward_basis=["DB","NL_ASSERTION"], even though the retained run evidence I audited does not show the listed calculate(...) tool call.

Examples:

• Task 38: the scenario asks whether the order can be brought under the card limit, and if not, what the most expensive item is and whether the order should just be cancelled. The listed action sequence includes calculate("466.75 + 288.82 + 135.24 + 193.38 + 46.66"). In the retained run evidence I audited, I still see the correct camera / $481.50 information and the order ends up cancelled, but I do not see the listed calculate(...) action.
• Task 45: the scenario asks to exchange a robotic vacuum cleaner and report the price difference. The listed action sequence includes calculate("652.61 - 642.72"). In the retained run evidence I audited, the exchange call is preserved and the final $9.89 message is preserved, but I do not see a retained successful calculate(...) call. I saw this more than once on task 45.
• Task 61: the scenario asks to change wireless earbuds in order W5061109 to a blue variant as long as the price is the same or lower. The listed action sequence includes calculate("256.67 - 226.49"). In the retained run evidence I audited, the final modification and post-run order state match, but I do not see the listed calculate(...) action.

I do not know whether calculate is intended to be a mandatory tool call here or just one possible way to reason about the price difference.

Potential locations:

• data/tau2/domains/retail/tasks.json, for the task-level evaluation_criteria.actions
• src/tau2/evaluator/evaluator.py, because final reward only uses the components listed in reward_basis
• src/tau2/evaluator/evaluator_env.py, if generic/action tool checks are intended to be part of the rewarded contract

Possible fixes depend on intent:

• if calculate is supposed to be mandatory, should these tasks include ACTION in reward_basis or otherwise gate success on the action checks;
• if calculate is not supposed to be mandatory, would it make sense to remove it from the listed evaluation_criteria.actions for these tasks so the task specification matches the scoring contract.

Could you share which interpretation is intended?

If helpful, I can attach a small evidence package with the task JSON, reward info, and task-log excerpts for these runs.

[gss10282023]

These reports are part of our broader audit in https://arxiv.org/abs/2605.10448. If broader investigation or follow-up coordination would be helpful, please contact us at liyi.zhou@sydney.edu.au or sgao5239@uni.sydney.edu.au