Does ‘Failed to decode tool call’ usually happen?

[karenyun]

Hi, thanks for the great work!

When I run the RL code, I often see the error “Failed to decode tool call” in the log. Is this normal?

[ZhaochenYu0201]

Hi, thanks for your interest!
Yes, this is completely normal. The "Failed to decode tool call" error occurs when the model generates improperly formatted JSON, typically due to complex code structures or incorrect escape characters.
This is actually part of the training. These errors are fed back to the agent as learning signals to improve its tool-calling and self-correction abilities. You should see the frequency decrease as training progresses, since the RL process specifically trains the model to generate properly formatted tool calls.
Occasional errors are expected and beneficial for learning.

[karenyun]

Hi @ZhaochenYu0201,

Thank you so much!
I encountered another error during training. It appeared in code_math.py, specifically in the compute_score function where ast.parse was called. Is this a normal issue?

Another question — since I used multiple nodes, there was some abnormal behavior with the RewardManagerWorker. I’m not sure whether this issue also occurs when using a single node. And I’m also not sure whether this issue will affect the performance.

[Tianshi-Xu]

Hi, thanks for your interest! Yes, this is completely normal. The "Failed to decode tool call" error occurs when the model generates improperly formatted JSON, typically due to complex code structures or incorrect escape characters. This is actually part of the training. These errors are fed back to the agent as learning signals to improve its tool-calling and self-correction abilities. You should see the frequency decrease as training progresses, since the RL process specifically trains the model to generate properly formatted tool calls. Occasional errors are expected and beneficial for learning.

Hi Zhaochen,

I’ve successfully reproduced the results from the paper — thanks for making the code so clear!

I noticed that when the JSON format is incorrect, the model doesn’t seem to receive explicit feedback about the formatting error. I’m curious — where does the improvement in tool-use ability mainly come from in this case?

Also, I was wondering about the design trade-off here: adding explicit feedback for JSON formatting errors might help the model learn more structured tool calls, but it could also consume valuable output tokens and reduce efficiency. How did you balance this trade-off during training?

Thanks again for your time and clarification!

[ZhaochenYu0201]

Hi @karenyun , thanks again for the detailed report!

Yes – in our setting (multi-turn agent RL with code generation) these occasional errors from code_math.py are expected and, by design, they do not break training. They simply correspond to trajectories where the agent produced invalid / non-verifiable code and therefore receive a negative reward.

Specifically

1. ast.parse error in compute_score

   The traceback

   SyntaxError: unterminated string literal (detected at line 24)
   

   is raised when the agent’s generated Python code is syntactically invalid (e.g., unmatched quotes in a print statement). In code_math.compute_score we first do

   tree = ast.parse(solution)

   to sanity-check the code. If parsing fails, the SyntaxError is caught and the function returns something like

   {"score": -1.0, "acc": False, "pred": None}

   So this is normal in multi-turn agent RL: sometimes the agent produces bad code or code that doesn’t follow the expected format, and those samples are just penalized.

2. RewardManagerWorker ... 'NoneType' object is not subscriptable

   The repeated lines like:

   (RewardManagerWorker pid=...) 执行异常: 'NoneType' object is not subscriptable

   come from the reward function running inside the RewardManagerWorker, not from VerL’s core itself. They usually happen when:

   • We call check_correctness(...) to run the code in the sandbox, and
   • For some failing runs the returned metrics object is None or does not have the expected nested structure, but
   • The reward code still tries to index it as metrics[1][0]['api_response']['run_result']..., which triggers 'NoneType' object is not subscriptable.

   This exception is caught inside compute_score and mapped to a negative reward (with a log message). So again, it only marks that this particular trajectory/code execution failed; it doesn’t crash the trainer.

3. Multi-node vs single-node & impact on performance

   • The behavior is not specific to multi-node training. On a single node you can also see these messages; with multiple workers they just appear more often / interleaved in the logs.
   • As long as the reward function always returns a valid value (which it does after catching these exceptions), training proceeds normally. The only effect is that those bad trajectories get low/negative reward.
   • If such errors are rare, they won’t hurt performance. If you see them very frequently, it’s a signal that the agent is generating too much invalid code, and improving the prompting / decoding or adding stronger code-format constraints will help.

If you’d like to make the logs quieter or more robust, you can add extra None / shape checks around the metrics[...] indexing, but functionally these errors are expected in this kind of agent-RL setup and are safe to ignore unless they dominate your trajectories.

---

[ZhaochenYu0201]

@Tianshi-Xu
Hi, great question and glad to hear you reproduced the results!
Regarding feedback mechanism:
We return error messages for correctly parsed tool calls that fail during execution. However, for pure JSON parsing errors, we deliberately don't return feedback. Here's why:

1. JSON error messages often contain the original malformed code, which can be quite lengthy
2. Even with full error details, models rarely correct formatting issues in one attempt
3. After multiple correction rounds, accumulated error messages quickly bloat the context window, degrading performance or causing truncation

Where does improvement come from?
We believe the model learns this through RL in an emergent way. When the model receives error feedback, it knows the code format is correct but execution failed. When it receives no feedback at all (neither error nor result), it gradually learns to infer that a formatting issue occurred. This resembles an "aha moment" of capability emergence. Our experiments confirm this: tool call efficiency consistently improves throughout training.

Regarding the trade-off:
We haven't formally studied this yet, but you could experiment with:

1. Returning explicit but length-limited error signals
2. Using a standardized short message like "JSON parsing error detected. Please check for escape character issues and regenerate properly formatted code."

These approaches might help without overwhelming the context window. It would be interesting to see if they improve learning!

[karenyun]

Hi @ZhaochenYu0201, thanks so much for your response.
I trained the RL model based on your SFT model downloaded from Hugging Face. However, I observed an unusual increase in the actor’s entropy. Since I have deployed SandboxFusion locally, I am wondering whether the failed decoded tool calls might be contributing to this entropy increase.

[karenyun]

Hi @Tianshi-Xu,
Did you use the Sandbox API or a locally deployed Sandbox?

If you used a local deployment, could you share how you set up your Python environment?
Using python=3.10 causes an issue, but switching to python=3.11 leads to other problems during training.

If you used the API, could you share your API settings? I always got the error API call failed.
Thanks so much!

[ZhaochenYu0201]

Hi @karenyun, thank you so much for your interest in our work!
Regarding the entropy increase you observed, I think your analysis is quite reasonable. When the model invokes tools and receives feedback from the external environment, this information is essentially unknown/uncertain to the model at generation time, which naturally leads to higher entropy at these positions. Additionally, failed tool calls—whether due to code parsing errors or deployment issues preventing correct code execution—will increase the model's uncertainty about subsequent decisions.
Therefore, tool call failures caused by external factors (such as sandbox environment deployment issues) will inevitably impact training. This is why we believe having a stable and robust code execution sandbox environment is crucial for RL training.
Regarding SandboxFusion deployment, as mentioned in our README, there are two ways to create a sandbox:

1. Deploy Sandbox Fusion locally (refer to the SandboxFusion deployment documentation)
2. Use Volcano Engine Cloud FaaS service (refer to Volcano Engine Code Sandbox documentation)

With either method, you'll obtain an API endpoint like https:///run_code, which should be configured in sandbox_fusion_tool_config.yaml.
In our experiments, we deployed it on the cloud as a FaaS service. Compared with local sandbox deployment, the cloud-based approach offers:

• Higher performance and reliability
• Significantly reduced training time overhead for code execution
• Decoupling from the training cluster, preventing potential damage to the local training environment from incorrect code

I would recommend checking your local SandboxFusion deployment to ensure it's stable and can handle concurrent requests reliably during training. If the issues persist, you might consider using the cloud FaaS service as an alternative.

Hope this helps! Feel free to reach out if you have more questions.

[karenyun]

Hi @ZhaochenYu0201, thanks so much for your patient and response.
I have depolyed the cloud FaaS service based on this link 'https://www.volcengine.com/docs/6662/1585066?lang=zh', but I’ve been running into the error:: Sandbox API call failed. ... Read timeout 70s.

I’m not sure whether this is caused by a misconfiguration on my side, or if you’ve encountered a similar issue before.
Much appreciated for your help!