terminal-rl: Terminal-Bench/SWE-RL cookbooks, custom losses, gateway & sandbox reliability (WIP)#740
Draft
jeffreysijuntan wants to merge 51 commits into
Draft
terminal-rl: Terminal-Bench/SWE-RL cookbooks, custom losses, gateway & sandbox reliability (WIP)#740jeffreysijuntan wants to merge 51 commits into
jeffreysijuntan wants to merge 51 commits into
Conversation
…uation mangling (#656) * fix(sandbox): add [environment].replay_dockerfile toggle + fix RUN-continuation mangling Non-docker backends (modal/daytona) replay a task's Dockerfile RUN steps on top of the booted image. Two defects (#655): 1. Multi-line `RUN ... \` steps were rejoined with "\n" instead of a space, producing invalid shell (observed: `bash: syntax error near '&&'`). 2. The replay assumes the configured image is a *base*. Terminal-bench / Harbor tasks boot a *fully-built* image, so replaying the RUN steps double-applies the build (`git clone ... already exists`, missing COPY'd files, etc.). Fixes: - `_dockerfile_run_commands` joins `\`-continuations with a space so multi-line RUN steps stay valid when re-exec'd via `bash -c`. - New `[environment].replay_dockerfile` toggle. Default `true` preserves SWE-bench base-image behavior; `false` skips replay for prebuilt task images. Honored across the resolution cold path, daytona + modal snapshot builds, and the snapshot env-key fingerprint. - Tests for the continuation join, COPY skipping, and the toggle (incl. the task.toml -> metadata path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: apply ruff-format to satisfy pre-commit Collapse multi-line expressions onto single lines per ruff-format (line-length=200), fixing the failing pre-commit check on PR #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_config() rebuilt config.json from scratch, writing only provider/model/api_keys/base_url. Any other key was dropped — so `rllm model swap`/`setup` silently wiped the `ui_api_key` written by `rllm login`, logging the user out of the rLLM UI. Merge into the existing file instead (mirroring save_ui_config). Adds a regression test. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#658) * feat(gateway): cumulative token mode for the in-process (Tinker) path `cumulative_token_mode` (drift-free multi-turn token forwarding: turn 2+ is rewritten to a pre-tokenized prompt built by renderers.bridge_to_next_turn, avoiding decode→re-encode drift) previously only worked for the HTTP-proxy (vLLM/verl) path — the cumulative handlers always routed to an HTTP worker's /v1/completions, so backends that run in-process via `local_handler` (Tinker) fell back to re-rendering + re-tokenizing the full conversation every turn. This wires the same feature through the local_handler path, reusing the existing backend-agnostic TokenAccumulator + Prime Intellect `renderers` bridge: - tinker_adapter: the handler now detects a pre-tokenized `prompt` (list[int]) and samples straight from it via the engine's existing get_token_output_from_token_input + assemble_model_output, returning a completions-style body (prompt_token_ids + choices[].token_ids) — the shape the gateway's cumulative handler already extracts token IDs from. The chat (messages) path is unchanged. - proxy: _handle_cumulative_non_streaming routes to local_handler when present (no HTTP worker); added _handle_cumulative_streaming_local to synthesize an SSE stream from the single in-process completion (mirrors _handle_streaming_local), with the same token ingest. Net effect: Tinker rollouts get the same prefix-extension guarantee verl has — turn N's prompt tokens are byte-for-byte prior turns' prompt+completion tokens, so the sequence the optimizer trains on matches what was generated. Tests (no Tinker service / model required): - tests/test_tinker_adapter_cumulative.py — token-prompt path samples from tokens; chat path unchanged; non-int prompt falls through. - rllm-model-gateway/tests/unit/test_cumulative_token_mode_local.py — proxy local cumulative non-streaming + streaming ingest + chat translation. Existing cumulative/accumulator suites (28) still pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(gateway): wrap long SSE chunk lines to satisfy ruff E501 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cookbooks): add swe-rl recipe (rllm-swesmith → SWE-bench Verified)
End-to-end SWE-RL cookbook that pairs rLLM's native `rllm-swesmith`
training set with `harbor:swebench-verified` for eval, driving the
in-tree `mini-swe-agent` harness inside per-task sandboxes. Default
model is Qwen/Qwen3.5-9B + LoRA-32, GRPO + async + compact filtering,
64 parallel Daytona sandboxes.
No custom AgentFlow or evaluator: the harness owns the action loop,
each task's `tests/test.sh` is the verifier (pytest for swesmith, the
official SWE-bench harness for Verified), and the gateway captures
trajectories transparently.
Files: prepare_data.py (dataset pull), train.py + train_{tinker,verl}.sh
(recipes), test.py (catalog + harness smoke tests), README.md, pyproject.toml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cookbooks/swe-rl): use rllm.rollout.* keys; drop stale gateway overrides
The unified trainer config exposes sampling params at
`rllm.rollout.{train,val}.{temperature,top_p}`. The `sampling.*` paths
copied from `examples/harbor_swe/train_harbor.sh` predate the unified
config and break Hydra struct validation. Same template also passed
`rllm.gateway.public_url` / `rllm.gateway.sampling_params_priority`,
neither of which exist in the current schema — drop them.
Verified by resolving the config with --cfg job: no validation errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(swe-rl): run on rLLM-native SandboxedAgentFlow path; unblock async training
Switch the cookbook off the remote Harbor runtime onto rLLM's own
SandboxedAgentFlow path (AgentFlowEngine) and fix the bugs that surfaced
once real rollouts ran end-to-end on Modal sandboxes.
train.py / train_*.sh:
- Pass MiniSweAgentHarness as agent_flow so AgentTrainer auto-wires
SandboxTaskHooks + per-task verifiers via AgentFlowEngine (not the
remote_runtime/RemoteAgentFlowEngine path). Sandbox backend selected by
SWE_SANDBOX_BACKEND (default modal).
- Load the val split as "default" (Harbor-pulled name), not "test".
- Async training requires train_batch_size=1 and raise_on_error=false; set
both. Effective batch is mini_batch_size(16) groups x group_size(8).
- Add SWE_VAL_MAX to cap the 500-task SWE-bench-Verified val set.
- Drop stale rllm.remote_runtime.* overrides; doc the sandbox backends.
rllm/data/utils.py:
- task_from_row now roots the Task at the row's task_path and merges
task.toml/Dockerfile metadata, so per-task verifier + image resolution
work on the training path (fixes "No verifier configured").
rllm/gateway/tinker_adapter.py:
- Translate TerminationEvent into an OpenAI-standard 400
context_length_exceeded instead of a 500. litellm maps it to a
non-retryable ContextWindowExceededError, so an over-length in-sandbox
agent stops immediately instead of retrying until the run-timeout
SIGKILL — which was stalling group completion and wedging async steps.
rllm/harnesses/mini_swe_agent.py:
- Retry the in-sandbox uv install (Modal->GitHub egress resets the
connection intermittently and aborted the install under set -e).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(gateway): cumulative token mode for the in-process (Tinker) path
`cumulative_token_mode` (drift-free multi-turn token forwarding: turn 2+ is
rewritten to a pre-tokenized prompt built by renderers.bridge_to_next_turn,
avoiding decode→re-encode drift) previously only worked for the HTTP-proxy
(vLLM/verl) path — the cumulative handlers always routed to an HTTP worker's
/v1/completions, so backends that run in-process via `local_handler` (Tinker)
fell back to re-rendering + re-tokenizing the full conversation every turn.
This wires the same feature through the local_handler path, reusing the
existing backend-agnostic TokenAccumulator + Prime Intellect `renderers`
bridge:
- tinker_adapter: the handler now detects a pre-tokenized `prompt` (list[int])
and samples straight from it via the engine's existing
get_token_output_from_token_input + assemble_model_output, returning a
completions-style body (prompt_token_ids + choices[].token_ids) — the shape
the gateway's cumulative handler already extracts token IDs from. The chat
(messages) path is unchanged.
- proxy: _handle_cumulative_non_streaming routes to local_handler when present
(no HTTP worker); added _handle_cumulative_streaming_local to synthesize an
SSE stream from the single in-process completion (mirrors
_handle_streaming_local), with the same token ingest.
Net effect: Tinker rollouts get the same prefix-extension guarantee verl has —
turn N's prompt tokens are byte-for-byte prior turns' prompt+completion tokens,
so the sequence the optimizer trains on matches what was generated.
Tests (no Tinker service / model required):
- tests/test_tinker_adapter_cumulative.py — token-prompt path samples from
tokens; chat path unchanged; non-int prompt falls through.
- rllm-model-gateway/tests/unit/test_cumulative_token_mode_local.py — proxy
local cumulative non-streaming + streaming ingest + chat translation.
Existing cumulative/accumulator suites (28) still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): enable cumulative token mode + raise context window
Enables the in-process cumulative-token path (cherry-picked from #658) in the
tinker recipe for testing: rllm.gateway.cumulative_token_mode=true +
renderer_family=qwen3.5. Also raises training.max_length=65536 /
data.max_prompt_length=57344 so long mini-swe-agent trajectories fit, and
save_freq=10.
NOTE: the cumulative feature commit on this branch is a cherry-pick of #658 for
local testing — drop it (rebase onto main) once #658 merges to avoid duplication.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): add synchronous tinker training script
train_tinker_sync.sh: on-policy variant of train_tinker.sh for testing —
drops async_training (synchronous generate→train) and uses a real
data.train_batch_size (default 4; effective batch = train_batch_size x
group_size). Same model/sandbox/context/cumulative settings otherwise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): disable cumulative token mode (incompatible with thinking model)
Qwen3.5 is a thinking model (<think>...</think> per assistant turn). The normal
re-render path strips prior-turn reasoning from history (mini-swe-agent stores
the think-stripped message.content), but cumulative mode carries forward the RAW
completion tokens (tinker_engine completion_ids=response_tokens, incl. <think>),
which renderers.bridge_to_next_turn concatenates verbatim. The model then sees an
ever-growing stack of its own prior reasoning — out-of-distribution for multi-turn
— degrading into short, submit-early, zero-reward trajectories.
Renderer mismatch was ruled out: the bridge (PrimeIntellect qwen3.5) and the
engine renderer are token-identical; the bridged prompt is well-formed. The
issue is purely reasoning carry-forward. Cumulative mode is fine for
non-thinking models / single-turn; for thinking models, correct multi-turn
rollout requires stripping prior reasoning (i.e. re-tokenization), which is
fundamentally at odds with exact-token reuse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): switch to Qwen3.5-4B + terminus2 harness
- agent_flow -> Terminus2Harness (was MiniSweAgentHarness) in train.py;
docstring/script prose updated accordingly.
- model.name / MODEL_PATH -> Qwen/Qwen3.5-4B; renderer_family -> qwen3.5;
experiment names -> r2egym-terminus2-qwen3.5-4b[-sync/-verl].
- re-enable rllm.gateway.cumulative_token_mode in the tinker scripts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(swe-rl): bump n_parallel_tasks to 128 (tinker + sync)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add train_fireworks.sh Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * Support FireworksEngine in gateway Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * add max_prompt_length limit Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * use tokenizer_model when exists. Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * style(gateway): fix ruff formatting (trailing whitespace) Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> * feat(swe-rl): scale Fireworks rollout deployment to 4 replicas Set fireworks_config.rollout_deployment_replica_count=4 in both the async and sync Fireworks scripts (the prior fireworks_config.replica_count key does not exist in the backend config). Co-Authored-By: 1stprinciple <ztyharbin@gmail.com> --------- Co-authored-by: 1stprinciple <ztyharbin@gmail.com>
…ks (#668) * feat(algorithm): add ECHO env-observation loss across verl/tinker/fireworks ECHO (arXiv:2605.24517) augments GRPO with an auxiliary cross-entropy loss on environment-observation tokens — the tool/terminal output the policy conditions on but that GRPO masks out of its loss. Advantages are identical to GRPO; the only change is the extra loss term, so every rollout (including failures) becomes dense supervision at no extra rollout cost. Selectable via the unified trainer on all three backends: rllm.algorithm.adv_estimator=echo # lambda defaults to paper's 0.05 rllm.algorithm.env_loss_coef=<lambda> # override; 0.0 == plain GRPO Shared core (backend-agnostic): - rLLMAdvantageEstimator.ECHO + env_loss_coef on AlgorithmConfig; None resolves to 0.05 for echo, else 0.0 (sentinel distinguishes unset from explicit 0). - echo registered as a GRPO-advantage alias in the estimator registry. Per-backend env-CE term (the only backend-specific part): - verl: added in CustomPPOLoss, reusing the existing forward pass log-probs and GRPO's loss_agg_mode/global-batch normalization. Single pass, exact, free. - tinker: gradient-accumulated cross_entropy pass (weights=lambda on obs tokens), returned separately so policy-pass logprobs stay aligned. Clean because tinker's optim_step applies no extra normalization. - fireworks: same second pass, but normalization is folded into weights/advantages and GradAccNormalization is forced to NONE (its counting normalization spans passes and would otherwise rescale the policy gradient). Defaults to 0.0 everywhere, so existing GRPO runs are unchanged. Adds ECHO config tests and documents the switch in the swe-rl cookbook README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(design): auxiliary-loss framework (AuxiliaryLoss) + ECHO/SDAR clients Design doc for a general token-level auxiliary-loss abstraction so algorithms like ECHO and SDAR are a registered spec + config instead of bespoke per-backend surgery. Covers: named token masks (data layer), the AuxiliaryLoss interface (mask/target/weight/requires), a registry, the aux_losses config list, the two backend executors (in-process for verl; gradient-accumulated cross_entropy pass for tinker/fireworks), ECHO and gap-gated SDAR as reference clients, the verl/managed capability model with graceful degradation, and an incremental migration plan that ports the just-merged ECHO onto the framework first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): generalize ECHO into an AuxiliaryLoss framework Prototype of the auxiliary-loss abstraction from design/auxiliary-losses.md (migration steps 1-3): token-level auxiliary losses become a registered spec + config instead of bespoke per-backend code, and ECHO is ported onto it as the first client. Behavior-preserving for ECHO. Shared core (rllm/trainer/algorithms/aux_loss.py): - AuxiliaryLoss spec (mask / weight / requires), register_aux_loss registry, EnvPredictionLoss, and build_aux_losses() with env_loss_coef / adv_estimator=echo back-compat sugar. New algorithm.aux_losses config list (a stack of {type, coef, ...} specs). Backend executors (the only backend-specific part): - verl: CustomPPOLoss._apply_aux_losses folds each term into the single existing forward/backward pass (replaces _add_env_loss). - tinker: _get_aux_loss_futures submits one gradient-accumulated cross_entropy pass per loss (replaces _build_env_datum/_get_env_loss_futures). - fireworks: _build_aux_datums + the normalization fold / GradAccNormalization.NONE now driven by build_aux_losses() rather than env_loss_coef. - Shared managed datum builder in rllm/trainer/tinker/aux_loss.py (aux_positions + build_aux_ce_datum), used by both tinker and fireworks. Adding a new aux loss is now one registered class + config, no per-backend surgery. SDAR's dynamic-weight + teacher-forward extensions (the `requires` field) are the next milestone and currently raise NotImplementedError. Tests: tests/unified_trainer/test_aux_loss.py covers the registry, config resolution + ECHO sugar, the managed datum builder, and the executor math (incl. empty-observation safety). ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(tinker): satisfy ruff UP038 in aux_loss isinstance Use `int | float` instead of `(int, float)` in the isinstance call (rllm/trainer/tinker/aux_loss.py:51) to fix the pre-commit ruff failure on PR #668. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(algorithms): rewrite RL algorithms overview + register in nav (#675) The core-concepts/rl-algorithms page was stale (documented an old kl_ctrl/AgentPPOTrainer API, only GRPO/PPO/ReMax) and unlinked from the nav. Rewrite it as an accurate overview of the current algorithm surface: - built-in advantage estimators (grpo, reinforce, reinforce_plus_plus_baseline, prpo, rloo, echo) with exact adv_estimator strings - ECHO + the auxiliary-loss framework (env_loss_coef / aux_losses, per-backend cost, metrics) - how to build custom RL algorithms: custom advantage estimator (@register_rllm_adv_estimator) and custom auxiliary loss (@register_aux_loss) - config quick reference + cross-links to advantage-estimator / configuration / capability-matrix / backends Registers the page under the "Advanced algorithms" nav group. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#674) End-to-end terminal-agent RL recipe, sibling of cookbooks/swe-rl: - train on a local set of Harbor-format terminal-agent tasks (a .tar.zst you provide via TB_TRAIN_TARBALL), eval on harbor:terminal-bench@<TB_EVAL_VERSION> (default 2.0; flip to 2.1 once published) - terminus2 harness runs inside per-task sandboxes; per-task tests/test.sh is the verifier (writes /logs/verifier/reward.txt) - tinker / tinker-sync / verl / fireworks / fireworks-sync train scripts - README documents the ECHO switch (rllm.algorithm.adv_estimator=echo) The training dataset is not public yet, so the README/scripts describe it generically and take the tarball path from TB_TRAIN_TARBALL. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#677) * feat(harness): configurable rollout limits — uncapped turns, longer timeout, Modal lifetime headroom Stop cutting sandboxed CLI rollouts short, and make the limits configurable rather than hardcoded: - terminus2: max_turns defaults to None (no artificial cap — Harbor's own default) instead of 50. RLLM_TERMINUS_MAX_TURNS is emitted only when set, and the in-sandbox driver treats an absent value as uncapped. - BaseCliHarness: per-rollout run_timeout default 1800 -> 3600s; configure() now consumes an `agent_timeout` override. - eval CLI: add `--agent-timeout SECONDS` (routed through configure(); also published as RLLM_HARNESS_RUN_TIMEOUT_S so the sandbox lifetime tracks it). - modal_backend: derive the sandbox lifetime as run_timeout + install_timeout + headroom so the container can't be reaped before the agent's own timeout fires — the exit-137 / "Sandbox already shut down" failure. The lifetime clock starts at create, before install + run + verify, so a flat default collided with the run timeout. RLLM_MODAL_SANDBOX_TIMEOUT_S still overrides. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cookbooks): wire TERMINUS_MAX_TURNS knob + tune terminal-rl fireworks config - terminal-rl & swe-rl train.py: read the TERMINUS_MAX_TURNS env knob (empty/0 = uncapped) and pass it to Terminus2Harness(max_turns=...). - all train_*.sh: default TERMINUS_MAX_TURNS=100 (overridable per run), so the operational cap lives in the scripts rather than the harness. - terminal-rl/train_fireworks.sh: grow the context window to match longer, uncapped rollouts — training.max_length 64k->128k, data.max_prompt_length 56k->120k, enable gateway.cumulative_token_mode, and lower async staleness_threshold to 0.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Surface the loaded flow's effective knobs as a "Config" row in the `rllm eval` header (alongside Benchmark / Model / Agent / Evaluator): max turns, temperature, run/install timeouts, max concurrent. Knobs are read off the agent instance via a curated (attr, label, formatter) spec, so a flow that doesn't expose one simply omits that chip and an agent exposing none gets no row — graceful across harnesses (react, CLI agents) and harbor runtimes. Rendered as dim-key / bold-value chips joined by a dim "·"; spaces within a chip are non-breaking so a pair like "run timeout 3600s" never splits across a line wrap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(fireworks): per-trajectory session affinity + prompt-cache metric Fireworks reuses a replica's prompt-prefix KV only when a trajectory's turns route to the same replica (keyed on a session id via request headers). The rollout path set no such header, so multi-turn requests scattered across replicas and re-prefilled the (growing) prompt every turn. (a) Session affinity: - Gateway threads the per-rollout session id into the in-process handler body (rllm_session_id) at all local-handler call sites. - tinker_adapter forwards it only to engines advertising supports_session_affinity (Fireworks), so the tinker path is unchanged. - FireworksEngine sets x-multi-turn-session-id / x-session-affinity per request. Since DeploymentSampler.async_completions_stream has no per-call header hook, _inference_headers is monkeypatched to merge headers stashed in an async-safe ContextVar set around each call. (b) Prompt-cache metric: - _record_prompt_cache accumulates ServerMetrics.cached_prompt_tokens / prompt_tokens, adds a per-call prompt_cache_hit_ratio to the metrics dict, and logs a rolling hit-ratio every 50 completions. Low ratio => affinity not pinning a trajectory to one replica. Both are automatic — no config change. Verify by watching the "Fireworks prompt cache: hit_ratio=..." log climb across a rollout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(fireworks): rename session headers + drop in-process prompt-cache recording - Rename affinity_headers -> session_headers (the per-request x-multi-turn-session-id / x-session-affinity headers). - Drop the in-process prompt-cache recording (_record_prompt_cache, the rolling counters, and the per-call prompt_cache_hit_ratio metric); read cache behaviour from the Fireworks console instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…n the picker Two CLI UX fixes for the Fireworks eval path. Sampling params: - `rllm eval` now seeds temperature=1.0, top_p=1.0 (mirrors the cookbook training rollout), overridable by --sampling-params / --temperature / --top-p and always shown in the header's gateway-enforced "Sampling" row. - Removed the duplicate temperature from the "Config" panel — it showed the harness's requested value (0.7) while the gateway enforced a different one (e.g. 1.0), so the same param appeared twice with conflicting values. - Terminus2Harness default temperature 0.7 -> 1.0 (and the driver fallback) so nothing lingers at 0.7 when sampling isn't gateway-enforced. Fireworks deployment fetch: - fetch_fireworks_deployed_models only queried the deployedModels endpoint (populated when a custom/fine-tuned model is bound to a deployment), so a dedicated deployment of a base model — which has no deployedModel entry and is addressed for inference by its accounts/<acct>/deployments/<id> name — was invisible in the `rllm setup` picker, even though that's the exact id evals run against. Now also lists READY deployments by their id (e.g. gpdl1exc). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The agentflow_engine tqdm bar only updated the completed count; add a set_postfix with the cumulative mean episode reward so the current score is visible live during a run instead of only per-rollout lines. Falls back to is_correct for error/empty episodes. Display only. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… metrics, prompt-cache affinity
Terminal-bench eval on Modal scored ~30-33% vs official Terminus-2 ~40%, with
flaky "Command failed" warnings. Root cause was infra, not model quality.
The Modal sandbox lifetime was derived from a global env var (a flat 4800s),
while the agent runs for the per-task agent_timeout (600-12000s) and the
verifier then runs in the same box. 14 of 89 terminal-bench@2.0 tasks have
agent+verifier > 4800s, so the box was reaped before the verifier's exec
("Sandbox already shut down") -> the whole episode became a 0-step ERROR
(all agent work discarded). With retry_limit=1, each became a permanent zero.
Changes:
- eval/_resolution: size the Modal sandbox per-task, max(override,
agent_timeout + verifier_timeout + install + slack), so the box always
outlives the agent + verifier it hosts.
- eval/runner: eval retry_limit 1 -> 2, so transient infra failures get one
retry instead of becoming permanent zeros (matches the official harness).
- engine/agentflow_engine: live running accuracy/reward on the progress bar
(acc N/done=% reward=...) so the score is visible as the eval runs.
- sandbox/protocol + modal_backend: classify an exec killed at its own timeout
as SandboxCommandTimeout instead of a generic "Command failed (exit -1)";
cli_harness catches it and logs at INFO ("reached its time budget") — an
agent spending its full per-task budget is expected, the verifier still
scores the partial state.
- gateway proxy: inject x-session-affinity / x-multi-turn-session-id from the
rollout session id on the HTTP path, so eval multi-turn trajectories reuse
Fireworks' per-replica prompt cache. Previously only the in-process training
path set affinity; eval (HTTP proxy path) sent none.
Note: modal_backend.py also carries a pre-existing _CreateRateLimiter (Modal
create-rate pacing) that shares the new `import time`; it is bundled here
because it lives in the same file and is also eval-reliability related.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ask agent_timeout
BaseCliHarness.run() resolved the rollout exec timeout as
`task.metadata.get("agent_timeout", self.run_timeout)`, so a task carrying a
per-task agent_timeout always won and an operator-set
RLLM_HARNESS_RUN_TIMEOUT_S was only a fallback that never fired. Harbor tasks
always carry one — task_from_row merges task.toml's [agent].timeout_sec into
metadata — so e.g. training tb-opus-pass with RLLM_HARNESS_RUN_TIMEOUT_S=1800
silently ran rollouts to the task's 3600s budget. The intended wall-clock cap
was ignored.
Make run_timeout a hard ceiling when an operator sets it (env var or
--agent-timeout): effective timeout = min(per-task agent_timeout, run_timeout).
When it's unset, the task's own agent_timeout governs, so eval still honors each
benchmark's per-task budget (no regression to the terminal-bench parity work).
A run_timeout_is_cap flag (captured from the env at import, flipped on by
configure() for --agent-timeout) distinguishes the two.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…xy resilience (#684) * feat(sandbox/modal): isolate each run's sandboxes by run id Name the Modal App per-run (rllm-sandbox-<run_id>) and stamp each sandbox with a name + rllm_run_id tag, so on a shared account you can list/terminate just one run's (leaked) sandboxes via 'modal app stop rllm-sandbox-<run_id>'. run_id comes from RLLM_RUN_ID (sanitized) or a random per-process id (new rllm.env.rllm_run_id()). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(eval): proxy-death monitor, empty-completion canary, streaming UI uploads proxy.py: capture the litellm proxy's stdout+stderr (was DEVNULL) and add a monitor thread that loudly reports if the proxy dies mid-run — previously a dead proxy was silent and every rollout returned empty with no visible cause. agentflow_engine.py: flag rollouts whose LLM completions are ALL empty (upstream down — dead proxy / no healthy workers / model) and stream each enriched+scored episode to the UI as it finishes. runner.py + tracking.py: invoke on_episode_complete streaming (per-rollout) instead of an end-of-run burst; make the UI drain timeout env-tunable (RLLM_UI_DRAIN_TIMEOUT_S, default 600s). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * build(deps): bump fireworks-ai to 1.2.0a83 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): raise worker health-check timeout 5s -> 20s Under a concurrency spike the single litellm proxy worker is slow to answer /health while busy forwarding requests; a 5s timeout marked it dead (3 strikes) and route() then raised 'No healthy workers available', failing agent turns. 20s tolerates the transient slowness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eval): auto-pick free port for eval proxy so concurrent jobs don't collide The auto-started LiteLLM proxy hardcoded port 4000, so two concurrent `rllm eval` jobs collided on it — and the loser's startup probe could see the winner's proxy answering and then clobber its routing via /admin/reload. EvalProxyManager now binds a free port by default (proxy_port=None), with a retry-on-bind-race so concurrent launches each settle on their own port. Add --proxy-port to pin a deterministic port when debugging a single run. The gateway layer was already free-port, so port 4000 was the only collision point. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rLLM's native sandbox path (SandboxTaskHooks + rllm/sandbox backends, used when a non-`harbor:` agent runs a harbor-sourced dataset) reimplements task setup/exec instead of calling Harbor, and had drifted from how Harbor's Modal environment actually starts and configures a task. Bring it back into line: - Prebuilt images boot as-is. A task that declares its own [environment].docker_image ships a fully-built image; replaying its Dockerfile RUN steps on top double-applies the build. Default replay_dockerfile off in that case (matches Harbor's should_use_prebuilt_docker_image). SWE-bench-style tasks (no docker_image: base image + RUN extras) keep replay on. - Resource limits actually apply. Harbor task.toml uses size strings (memory = '4G'); rLLM only read memory_mb/storage_mb, so the limit was silently dropped to the Modal default. Normalize the string forms to *_mb. - Real user switching on Modal. exec() now applies `user` via `su <user> -s /bin/bash -c` (matching harbor.environments.modal) instead of ignoring it and running everything as root. - Task env reaches every exec. [environment].env is exported into every command (ModalSandbox.set_env), mirroring Harbor's per-exec Secret, instead of a one-shot `export` that didn't survive a fresh shell. - Keepalive on sandbox create, so the box stays alive regardless of the image's ENTRYPOINT/CMD (matches Harbor). Verifier/reward dirs locked away from agent_user are now chowned to the verifier's user (root unless a distinct verifier_user is set) so the now-su'd verifier can still write reward files. Tests: tests/eval/test_harbor_parity.py + tests/sandbox/test_modal_backend.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ion on the queued log line (#690) * feat(trainer): per-task datum/step/segment breakdown when a task is queued A task is consumed as one GRPO group, which prefix-merges into training rows; previously only batch-level merge metrics were visible. Log per-task at INFO when the group is finalized: number of groups, trajectories, steps (turns), and datums (rows after prefix-merge), plus steps/datum. _segment_count() applies the same prefix-extension check the backend transform uses, so the per-task datum count matches what's trained and the merge ratio is visible per task, not just batch aggregate (steps/datum == 1 is a clean cumulative trajectory; >1 a prefix break). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(trainer): add per-task group reward distribution to the queued log line Extend the per-task "Task ... queued" accounting (cherry-picked from feat/native-renderers) with the GRPO group's reward distribution: avg/min/max over its trajectories. Advantages are driven by within-group reward spread, so a min==max group carries no learning signal — surfacing this per task makes a degenerate group visible at a glance, alongside the steps->datums merge ratio. _traj_reward() mirrors the reward extraction in _log_prompt_group_finished (trajectory-level reward, falling back to the last step's reward) so the per-task average matches the per-episode rewards logged there. Unscored trajectories are excluded; an all-unscored group reports 0.0 without error. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Reproduce Ai2's Tmax recipe (arXiv:2606.23321) in rLLM. Adds: - rllm/data/tmax_builder.py: native builder for the TMax-15K corpus. The Harbor-registry copy their README cites (tmax/TMax-15K-Harbor) is not in the public Harbor registry, so build from Hugging Face instead — join allenai/TMax-15K (test_final_state pytest verifier + description) with allenai/tmax-15k-open-instruct (per-task prebuilt images) by task_id, and materialize Harbor task dirs. Reward = test_final_state.py pytest pass/fail -> /tmp/rllm/reward.json. Same pattern as r2egym/swesmith builders. - rllm/registry/datasets.json: tmax-15k catalog entry (builder-backed). - cookbooks/tmax/: prepare_data.py, train.py, train_verl.sh (faithful full-FT DPPO->GRPO + centered advantages, group 32, 65K, lr 1e-6, no KL, 64 turns), train_fireworks.sh / train_tinker.sh (LoRA variants), test.py, README. Sibling of cookbooks/terminal-rl. Builder join + materialization validated on real HF data; in-container reward execution needs Docker Hub image access. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#692) * feat(gateway): classify and explain TokenAccumulator resets in the log Cumulative-token mode previously logged a single ambiguous line on reset ("Resetting TokenAccumulator (was at turn N, M messages)") that collapsed four distinct causes and omitted the session id, so it was impossible to tell why a reset happened. A turn-1 reset in particular could be an upstream retry, a rewritten history, an assistant-only delta, or a renderer that can't bridge — all indistinguishable. Introduce a ResetReason taxonomy named by the observable structural relationship between the incoming request and the accumulated snapshot: - duplicate: incoming list identical to the last processed turn (conversation did not advance) — likely upstream retry - prefix_changed: snapshot-covered messages shrank or diverged (history compacted/edited, or session id reused) - empty_delta: clean extension, but only assistant message(s) added - renderer_no_bridge: valid extension the renderer declined A single plan_turn() classifier replaces the split is_cumulative / extract_new_messages logic at the proxy call site. reset() now logs at INFO with the session id, the machine-readable reason, a short plain-language explanation, reason-specific diagnostics (age_s, first_divergent_index, ...), and a reset_count that surfaces reset storms on one session. Switch the snapshot from a whole-list fingerprint to per-message fingerprints so prefix_changed can report the first divergent message and a short preview. Tests: per-reason classification + reset-logging coverage added; existing cumulative-mode unit + integration tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): add snapshot/incoming token counts to TokenAccumulator reset log The reset line now reports snapshot_tokens (prompt+completion of the last ingested turn = the accumulated history size right before the reset) and, for prefix_changed resets, incoming_tokens (best-effort token count of the incoming, typically compacted, request). Comparing snapshot_tokens against the model's context limit makes token-limit-triggered compaction obvious from the log, and incoming_tokens shows how far the history shrank. snapshot_tokens is free (the accumulator already holds the token IDs); incoming_tokens is computed via renderer.render_ids guarded so diagnostics can never break the reset path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(gateway): satisfy ruff-format on TokenAccumulator reset log string Join the reset log format string onto one line; ruff-format (pre-commit) rejected the manual two-line split. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…hful (#693) * feat(sandbox): make Daytona backend training-ready & Harbor-eval-faithful Bring the Daytona sandbox backend to parity with what e4e91cf (+#684) did for Modal, so TERMINAL_SANDBOX_BACKEND=daytona reproduces Harbor's Daytona eval performance and trains stably at GRPO scale. All Path-B execution semantics now match how Harbor's Daytona environment starts/configures a task. P0 — fidelity blockers (Daytona reward now matches Harbor): - Per-exec user switching: exec honors `user` via `su <user> -s /bin/bash -c` (resolving int UIDs via getent), so agent and verifier run under their declared users and the verifier-dir ownership setup in _resolution actually isolates. Mirrors modal_backend._build_exec_command and harbor's Daytona env. - SandboxCommandTimeout + guaranteed hard-kill: timed commands are wrapped in coreutils `timeout -k 10 <t>` (SIGTERM then SIGKILL) with the SDK timeout as a +60s backstop. Exit 124/137 and SDK-level timeout exceptions map to SandboxCommandTimeout, which cli_harness treats as "expected, score anyway" rather than a failure. A runaway agent can no longer hang the rollout. - set_env: declared [environment].env is re-exported into every exec (Daytona shells are independent), so the verifier reliably observes task env. P1 — training-scale stability: - Create retry + rate-limiting: a _CreateRateLimiter token bucket (RLLM_DAYTONA_SANDBOX_CREATE_RPS/_BURST) plus a _create retry loop with linear backoff on transient/rate-limit/5xx errors; a vanished snapshot / real validation error stays non-transient and cold-falls-back. Prevents GRPO's concurrent-create burst from losing rollouts. - Task-sized lifetime: _resolution sizes Daytona auto_stop_interval from agent+verifier+install+slack (RLLM_DAYTONA_SANDBOX_AUTOSTOP_MIN floor), so a long or stalled rollout isn't reaped mid-run. Also filter build_daytona_snapshot's Resources(**...) to compute-only keys (required for the new lifecycle kwarg; also fixes create_timeout previously leaking into Resources()). Tests: 22 new unit tests covering su/env wrapping, transient classification, the rate limiter, exec timeout->SandboxCommandTimeout (124/137, SDK exception, and the 124-without-timeout edge), and the create retry loop — all against fakes so no SDK/API key is required. Design doc at design/daytona-training-readiness.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(sandbox): unify sandbox lifetime knob into provider-agnostic RLLM_SANDBOX_TIMEOUT_S The sandbox-lifetime override was Modal-specific (RLLM_MODAL_SANDBOX_TIMEOUT_S), and the just-added Daytona path used a separate RLLM_DAYTONA_SANDBOX_AUTOSTOP_MIN. Two names for one concept ("how long may a sandbox live") is wrong now that the same training scripts can target either backend. Introduce a single provider-agnostic knob, RLLM_SANDBOX_TIMEOUT_S (seconds): - rllm/env.py: new sandbox_timeout_override_s() reads it, falling back to the deprecated RLLM_MODAL_SANDBOX_TIMEOUT_S (one-time warning) so existing scripts keep working. - _resolution.py: compute the per-task lifetime floor (agent+verifier+install+ slack) ONCE and raise it to the override, then each backend expresses it in its own unit — Modal's hard `timeout` in seconds, Daytona's idle `auto_stop_interval` in minutes. Dedupes the arithmetic that was copy-pasted across both branches. - modal_backend._default_sandbox_timeout(): read the shared helper. - Drop RLLM_DAYTONA_SANDBOX_AUTOSTOP_MIN (was unreleased, added in this branch). Adopt the new name in the clean tmax scripts (train_tinker.sh, train_verl.sh) and document it in both cookbook READMEs. The provider-specific create-rate knobs (*_CREATE_RPS/_BURST) stay separate — they tune different platform rate limits. Tests: provider-agnostic override floors both backends; legacy alias honored with canonical precedence; per-task sizing unchanged when no override is set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(sandbox): run Daytona sandboxes ephemeral by default RL rollout and warm-queue sandboxes are strictly single-use (create → run → verify → delete on close()), so default DaytonaSandbox to ephemeral=True with auto_delete_interval=0, matching how harbor's Daytona environment always runs. Two payoffs: - Higher quota: Daytona's ephemeral tier carries a larger per-sandbox limit, so tasks declaring more than the standard 10 GB disk (tmax declares 20 GB) now schedule instead of failing every create with "Disk request 20GB exceeds maximum allowed per sandbox (10GB)". This is why we honor the task's declared storage rather than clamping it — ephemeral lifts the ceiling. - Leak-proofing: an ephemeral sandbox is deleted (not left stopped) when it stops, so a SIGKILL'd run that never reaches close()/atexit no longer strands a stopped box forever — the failure mode behind the ~1.3 TB of leaked stopped sandboxes seen on the shared account. ephemeral is an overridable kwarg (=False falls back to auto_delete_interval=-1, never-delete). Tested via a fake daytona module so no SDK/API key is required. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#694) * fix(harness): restore TerminationReason on the SandboxedAgentFlow path CLI-harness rollouts (terminus2 / mini-swe-agent) always came back ENV_DONE, so compact filtering couldn't distinguish timeouts/errors/length-truncation and the policy was punished for infrastructure failures (a timed-out or sandbox-reaped rollout shouldn't train as a reward-0 sample). - BaseCliHarness.run() now returns an outcome Episode carrying the reason it observed: TIMEOUT on SandboxCommandTimeout, ERROR (+ details) on exec failure, None on a clean exit; stamps metadata["max_turns"] when the harness caps turns. Stays stateless (returns via value) so the shared-instance parallel rollouts are unaffected. - AgentFlowEngine grows derive_termination_reason() and replaces the blunt ENV_DONE fallback in _finish_episode: last finish_reason=="length" -> MAX_RESPONSE_LENGTH_EXCEEDED, turns at cap -> MAX_TURNS_EXCEEDED, else ENV_DONE. Only runs when the harness left it None, so harness-set TIMEOUT/ERROR win. - Relocate TerminationReason/TerminationEvent from rllm.workflows.workflow to rllm.types (they're used across engine, harnesses, eval, trainer — no longer workflow-specific). workflow.py re-exports them so existing imports keep working; first-party rllm/ import sites updated to the canonical location. MAX_PROMPT_LENGTH_EXCEEDED detection is deferred (TODO in derive_termination_reason). Tests: updated tests/harnesses/test_cli_harness.py (Episode return + TIMEOUT/ERROR /max_turns classification) and added tests/engine/test_termination_derivation.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refine: don't infer MAX_RESPONSE_LENGTH/MAX_TURNS on the CLI-harness path Reading harbor's Terminus-2 showed neither is recoverable from gateway traces: - finish_reason=="length" is not episode-terminal here. On a truncated response the agent appends an error and recursively re-queries the LLM (terminus_2.py _query_llm), so it continues — a trailing "length" is coincidental, not the cause. Combined with base.yaml's mask_max_response_length_exceeded=True, the old heuristic could silently drop otherwise-fine (incl. successful) episodes. - The trace count is LLM *calls*, not turns (retries, length-recovery re-queries and summarization inflate it), so n_steps>=max_turns can't reliably detect the cap; the true turn count lives only inside the sandbox driver. derive_termination_reason now returns ENV_DONE for clean exits; the meaningful signal is the harness-set TIMEOUT/ERROR (the actual fix). Drop the now-unused max_turns metadata stamping. Tests pin "trailing length is not terminal" as a regression guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refine: inline ENV_DONE fallback, drop trivial derive_termination_reason It only ever returned ENV_DONE; inline it in _finish_episode and delete its test. Tighten the harness comments/docstrings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…695) * feat(gateway): rllm tunnel CLI + ngrok backend + daemon auto-pickup Make stable-tunnel setup ergonomic and shareable. Free cloudflared quick tunnels get 429-throttled; ngrok with a reserved domain is stable but previously required a manual `ngrok http` in a side terminal + a hardcoded URL pinned per-script (only valid for one person's account). Adds an `rllm tunnel` CLI that runs a backend as a detached daemon and records its live URL to ~/.rllm/tunnel.json; training auto-discovers it with no per-run config and nothing personal in shared scripts. - gateway/tunnel.py: extract a shared _Tunnel base from CloudflaredTunnel; add NgrokTunnel (ngrok http --url --log json, actionable hints for authtoken/1-session/domain errors); create_tunnel() dispatch; spawn_detached() daemon + state-file helpers; resolve_auto_tunnel() ($RLLM_GATEWAY_TUNNEL > running daemon > cloudflared quick tunnel) - hooks.enable_gateway_tunnel: resolve via resolve_auto_tunnel and warn on quick-tunnel fallback (was a silent cloudflared default) - eval/config.py: load_/save_tunnel_config (per-user backend/domain/port) - cli/tunnel.py + main.py: new `rllm tunnel {setup,up,status,down}` group - cli/train.py: sandbox-routing header shows the resolved tunnel URL - base.yaml: document "ngrok"/"ngrok:<domain>" + daemon auto-pickup - tmax/train_fireworks.sh: drop hardcoded ngrok URL (now portable) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * update training scripts * cookbooks(terminal-rl): gateway port 9090 + drop hardcoded ngrok tunnel train_fireworks.sh now matches tmax: gateway on 9090 (the rllm tunnel daemon default) with no hardcoded rllm.gateway.tunnel, so it auto-discovers the tunnel via `rllm tunnel up`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff format (join implicit string concatenations) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
`rllm view` returned a 400 "Bad run id" for any run whose directory name contains '@' (e.g. terminal-bench@2.0) or '#' (model#deployment). The SPA fetches /api/runs/<id>/index with the id percent-encoded, but the handler validated the still-encoded string against a charset regex excluding '%'/'@'/'#', and never URL-decoded it before the filesystem lookup. Decode the run id + filename with urllib.parse.unquote before validation and lookup (both the /index and /episodes/<file> routes), and permit '@'/'#' in the safe-id/safe-file patterns. Path-traversal is still blocked by the ".."/"/"/"\\" guard and the _under_root check, both of which now run on the decoded value. Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ode bridge Adds rllm.renderers: one bridge-capable renderer interface (prime-rl's protocol) for every backend's models. get_renderer resolves prime-rl native -> TinkerRendererAdapter (wraps tinker_cookbook AND Fireworks-cookbook renderers, e.g. deepseek_v4) -> chat-template fallback. BridgingRendererMixin synthesizes bridge_to_next_turn for any deterministic renderer and is verified byte-identical to prime-rl's hand-tuned qwen3 bridge. - gateway: resolve the cumulative-mode renderer via rllm.renderers (+ renderer_name config). The chat-template fallback is best-effort (runtime prefix-check guards; drift resets, never corrupts) instead of a hard error. - FireworksEngine: opt-in unified renderer via renderer_name/renderer_family (the DeepSeek-V4 path); default keeps the chat-template path so existing Fireworks models do not regress. Tests: 48 pass (renderer + gateway), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… ChatTemplateParser (#702) * fix(fireworks): use unified renderer for turn-0 and skip ChatTemplateParser when pinned FireworksEngine built ChatTemplateParser unconditionally, whose construction eagerly runs apply_chat_template (verify_equivalence). Some served templates reject the probe messages — e.g. GLM-5.2 raises "'str object' has no attribute 'items'" — crashing engine init before training. The GLM script already pins RENDERER_FAMILY=glm-5, but it was only wired to the gateway; FireworksEngine never received it, so it fell back to chat_parser. Now: - fireworks_backend passes rllm.gateway.renderer_family to FireworksEngine, so the engine renders turn 0 with the same renderer the gateway uses for the cumulative bridge (turns 1+) — keeping them consistent. - FireworksEngine resolves the renderer first; when it pins a native (prime-rl or tinker) renderer it uses it and skips ChatTemplateParser entirely. Default (renderer_family=auto, no renderer_name) is unchanged — chat_parser as before. Test: pinned family uses the renderer and leaves chat_parser None; default still builds chat_parser. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(renderers): auto-detect Fireworks-cookbook models (GLM-5.2) without config The Fireworks cookbook ships renderer *implementations* (glm5, deepseek_v4, ...) but no model->renderer map, and tinker_cookbook's recommender doesn't know them, so nothing auto-routed GLM-5.2 to the cookbook renderer — you had to pin renderer_family=glm-5. - registry: resolve() now owns a Fireworks-cookbook prefix map (GLM-5.x -> glm5, DeepSeek-V4 -> deepseek_v4), consulted after prime-rl's exact-match map; imports training.renderer so cookbook renderers self-register; degrades gracefully if the cookbook isn't installed (never hard-fails). - FireworksEngine: resolve the renderer the same way the gateway does (always), so engine (turn 0) and gateway (turns 1+) pick the same renderer. Adopt it when pinned or when auto-detection lands on a cookbook renderer (e.g. GLM-5.2), and skip ChatTemplateParser; prime-rl models under plain auto keep chat_parser (no regression for working qwen runs). GLM-5.2 now trains with no renderer config: auto -> cookbook glm5 on both engine and gateway. Tests: cookbook prefix mapping + graceful fallback. 14 pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…(both backends) (#705) * fix(fireworks): parse completions via unified renderer in assemble_model_output Skipping ChatTemplateParser when a unified renderer is active left the inherited assemble_model_output asserting chat_parser is set (bypass_render_with_parser=True), crashing on the first response: AssertionError: chat_parser must be set when bypass_render_with_parser=True Override assemble_model_output in FireworksEngine: when a unified renderer is active, parse the completion via renderer.parse_response (ParsedResponse -> content/reasoning/ tool_calls); otherwise delegate to the inherited chat_parser path. Covers both turn 0 (get_model_response) and turns 1+ (the gateway token-prompt path), which both call it. Test: assemble_model_output parses via the renderer with chat_parser=None. 15 pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): make bypass_render_with_parser reflect renderer ownership bypass_render_with_parser was hardcoded True even when a unified renderer was active — a lie (True means "render/parse with ChatTemplateParser"), and the root reason assemble_model_output asserted on a None chat_parser. Set it to `self.renderer is None`: False when the renderer owns rendering+parsing, True when ChatTemplateParser does. Flipping the flag alone isn't sufficient — the parent's bypass=False branch is hardcoded to a tinker_cookbook renderer ((Message, term) parse_response), which the unified renderer (ParsedResponse) isn't — so the assemble_model_output override stays. This just stops the attribute from lying (and misleading completer.py etc.). Test: bypass_render_with_parser is False with a renderer, True without. 15 pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(engine): unify renderer handling in shared TinkerEngine for both backends Per review: the unified rllm.renderers handling belonged in the shared TinkerEngine, not a FireworksEngine-only override. Now both backends render+parse through one path. - TinkerEngine: add self.unified_renderer (default None) + a shared _render_prompt_token_input helper. assemble_model_output and the render path branch on the unified renderer first, then ChatTemplateParser, then the tinker_cookbook renderer (incl. VLM image chunks). Base behavior is unchanged (unified_renderer stays None for the Tinker backend), so chat_parser / tinker-renderer / VLM paths are preserved. - FireworksEngine: set self.unified_renderer via rllm.renderers.resolve and reuse the shared render+parse; drop the assemble_model_output override and the inline render. self.renderer stays None (the shared legacy/VLM branch is unreachable here). No external code reads rollout_engine.renderer, so introducing unified_renderer is safe. 15 renderer tests + 55 engine tests pass (the 1 fail / 4 errors are pre-existing env issues: verl not installed, torch.LongTensor). ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): honor explicit bypass_render_with_parser=True (escape hatch) FireworksEngine ignored the config's bypass_render_with_parser (it was swallowed by **kwargs) and always resolved a unified renderer. Make it a real parameter: when True, skip the unified renderer and use ChatTemplateParser — an explicit escape hatch, matching TinkerEngine (which already honors bypass=True via its unbuilt unified path). Default (False) keeps auto-resolution. Test: bypass_render_with_parser=True with renderer_family pinned -> unified_renderer None, chat_parser built, bypass True. 16 tests pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…the accumulator (#707) Minimal change vs terminal-rl. In cumulative-token mode an identical resend (upstream timeout/retry of a turn already processed) was classified as a reset: plan_turn returned action="reset" with ResetReason.DUPLICATE, so the proxy dropped the drift-free token state, re-tokenized from text as a fresh turn-0, and forced a spurious segment break for what is a single step. Handle DUPLICATE as a replay instead: regenerate from the same accumulated prompt and overwrite the current turn in place — do NOT reset. - token_accumulator: plan_turn returns action="replay" for DUPLICATE; ingest_turn(..., advance=False) overwrites the current turn without bumping turn_count. - proxy: handle() replays from acc.prev_prompt_ids via _handle_cumulative_turn (replay=True), plumbed through the cumulative handlers as advance=not replay. Logged once per duplicate at INFO (regenerating in place, no reset). A fresh sample (not a cached one) is returned, and each duplicate costs a real generation — so a retry that depends on resampling (e.g. terminus2's context-length summarization fallback) still makes progress, and the generation latency naturally rate-limits any retry loop. Coalescing concurrent in-flight duplicates is intentionally out of scope (rare; added complexity). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…faults (#708) A transient blip on the shared training client around the per-step weight hot-load (eager_parameter_sync_step=1) crashed entire multi-hour runs: the Fireworks SDK's forward_backward / optim_step / save_and_hotload have no retry ("failures propagate so the training loop can crash"), and a step-N forward_backward TimeoutError at step ~17 tore down the run. Wrap the per-step training-client RPCs (forward, forward_backward incl. aux, optim_step, save_and_hotload) in a retry helper that, on a transient error (timeout / connection / channel reset / 5xx markers), optionally reconnects the client to the SAME job and retries with linear backoff. Non-transient errors (bad data) and cancellation propagate immediately. - Reconnect refreshes the ReconnectableClient via the SDK's own _connect path (rlor_mgr.wait_for_existing(job_id) -> client._use_endpoint), re-attaching to the same job so server-side optimizer/gradient state is preserved. Used for forward/forward_backward/optim_step. save_and_hotload retries without reconnect (the WeightSyncer holds the raw client; re-dispatch is idempotent). - Config (fireworks_infra.common): step_max_retries (default 2, 0 disables), step_retry_backoff_s (default 10). Each retry logs at WARNING. CAVEAT (documented in _run_training_op): forward_backward / optim_step mutate server-side state. A retry assumes the failed RPC didn't commit it — true for channel/dispatch failures; a *false* timeout could double-apply for one step. Retries are kept low; this bounded, rare perturbation beats crashing the run. For genuine slowness, raise step_timeout instead. Adds tests/trainer/test_fireworks_policy_trainer_retry.py (transient classification, retry-succeeds, retry-exhausted, non-transient-not-retried, zero-retries-disables, reconnect on/off, reconnect-without-mgr no-op). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…rmat tasks Harbor-format tasks (e.g. honeycomb's AWS/LocalStack tasks) declare an [environment.healthcheck] that boots an in-image service (LocalStack) and COPY files (start_localstack.sh, ready.d seeds) into the image. Two gaps stopped them running on the native sandbox path: 1. No healthcheck handling — rLLM boots the container with `sleep infinity` and never runs the image CMD, so the service never starts. Add a presence-guarded _run_healthcheck step in SandboxTaskHooks.setup (after env materialization, before the agent) that execs the declared command on a bounded retry loop and raises on exhaustion. No-op when no healthcheck is declared (every non-service eval/training task), mirroring harbor's run_healthcheck. 2. COPY dropped on Daytona — the cold path pulls the Dockerfile's FROM base and replays only RUN steps, silently dropping COPY/ENV/WORKDIR. Build the real Dockerfile via daytona's Image.from_dockerfile for Dockerfile-based tasks (replay_dockerfile true) — the same primitive harbor's own Daytona environment uses — in both the cold path and build_daytona_snapshot. env_key now fingerprints the whole build context for these tasks so two tasks sharing FROM+RUN but differing in COPYed data (e.g. different ready.d seeds) don't collide on one snapshot / warm-queue sandbox. Also: a missing verifier reward file is now an explicit error (was a silent reward 0.0) so a broken verifier is distinguishable from an agent that genuinely scored 0, matching harbor's RewardFileNotFoundError. Verified E2E on remote Daytona with terminus2 on a honeycomb AWS/LocalStack task: the real image builds (COPY intact), the healthcheck boots LocalStack + seeds ready.d, and the verifier runs (6/7 tests pass; the miss is the agent, not infra). modal from_dockerfile is a tracked follow-up (untested here; _FROM_DOCKERFILE_BACKENDS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EzkMPi5tAr3e4T5HBQ14Dt
* feat(terminal-rl): classify infra/grading failures distinctly across eval + train
Wall-clock timeouts and grading-infra failures were collapsed into env_done /
a generic reward 0, so timeouts were mislabeled and broken verifiers trained as
real task failures. Mirror Harbor's granular failure taxonomy end-to-end.
Taxonomy (rllm/types.py)
- Add TerminationReason.{AGENT_SETUP_TIMEOUT, ENV_START_TIMEOUT, VERIFIER_TIMEOUT,
GRADING_ERROR, SANDBOX_ERROR}; keep TIMEOUT = agent wall-clock (reward stays valid).
- Add INFRA_ERROR_REASONS and a canonical exception-class -> reason map
(termination_reason_from_error), keyed on Harbor's class names. trial_helper's
map_termination_reason now delegates to it (single source of truth, both paths).
Agent timeout label (was the core bug)
- terminus2 driver self-limits agent.setup/agent.run with asyncio.wait_for and
raises Harbor's AgentSetupTimeoutError/AgentTimeoutError, then writes an
ExceptionInfo-shaped outcome sentinel and exits 0 (partial state still graded).
- BaseCliHarness.run reads the sentinel (authoritative over the | tee-masked exit
code); falls back to an elapsed-vs-budget backstop when none is present. Exec
gets a grace window so the sentinel write wins the race.
Grading errors (eval + train)
- EvalOutput gains a structured `error`; ShellScriptEvaluator tags verifier
timeout / missing / empty / unparseable reward / tests-dir upload, and
PythonModuleEvaluator tags verifier crashes, with Harbor-aligned names.
- _finish_episode promotes a grading error to the matching infra reason
(VERIFIER_TIMEOUT / GRADING_ERROR), overriding only non-infra outcomes; grading
stays unconditional so a timeout is still scored on partial state.
Filtering + error-rate tracking
- CompactFilteringConfig gains mask_{verifier_timeout,grading_error,sandbox_error,
agent_setup_timeout,env_start_timeout} (base.yaml defaults them True under the
existing master switch); agent TIMEOUT is deliberately not masked.
- Train: buffer records episode/infra_error plus the per-reason series.
- Eval: EvalResult.termination_breakdown + EvalItem.termination_reason; only infra
reasons count as errors (agent timeouts don't).
Tests for the sentinel/backstop classification and the evaluator error tagging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(harness): classify mid-run sandbox death as SANDBOX_ERROR via is_alive() probe
A dead sandbox and a benign non-zero CLI exit both surface as the same generic
exec exception, so the prior code marked both ERROR. Probe sandbox.is_alive()
in the failure handler: if the box is gone it's infra (SANDBOX_ERROR, reward
untrustworthy); a non-zero exit on a live box stays ERROR. The probe is guarded
(absent/raising is_alive falls back to ERROR), so non-probing backends are
unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(eval): show per-category termination breakdown in the eval summary
The results panel showed only an aggregate "Errors" count. Add a
"Terminations" row listing each termination_reason with its count
(infra-error reasons highlighted), so a flaky endpoint or broken verifier
is visible at a glance instead of hidden. Per-item termination_reason and
the granular error type are already persisted to results.json / rllm view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(metrics+log): drop "Rollout completed." prefix; add batch/infra_error parity
- agentflow_engine: the per-rollout line was noisy ("[uid] Rollout completed.
Rewards: ..."); drop the prefix, keep rewards/timing/termination.
- unified_trainer: add batch/infra_error (sum of infra-reason rates), mirroring
the buffer/async path's episode/infra_error so the sync trainer reports the
aggregate infra-failure rate too. Per-reason batch/termination_reason/* already
covers every reason via the enum loop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(terminus2): driver crashed on harbor 0.3.0 — define timeout classes locally + bump harbor to 0.13.2
Regression from the timeout-labelling change: the in-sandbox driver imported
`from harbor.trial.errors import AgentSetupTimeoutError, AgentTimeoutError`, but
the sandbox installs harbor 0.3.0 which has NO `harbor.trial.errors` module (it
was added later; the host venv runs 0.13.2, where I verified the import — wrong
version). So the driver ImportError'd at startup on every task → the agent never
ran → "No traces found" → reward 0, mislabelled ENV_DONE across the whole run.
Fix: define AgentSetupTimeoutError/AgentTimeoutError locally in the driver (only
the class *names* matter — the harness maps them to a TerminationReason via the
sentinel), so the driver no longer depends on that module and works on any harbor
version. Also bump the pinned harbor_version 0.3.0 -> 0.13.2 to match the host and
get a modern Terminus-2 (verified API-compatible: all ctor kwargs + setup/run
signatures + BaseEnvironment abstract methods line up). NOTE: the bump only takes
effect on a snapshot REBUILD (the install marker no-ops a baked snapshot); the
local-class fix is what unblocks the existing 0.3.0 snapshot immediately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(classify): MODEL_ERROR for downed-upstream rollouts; sandbox setup → SANDBOX_ERROR
- MODEL_ERROR: _finish_episode now detects "no usable model output" — the agent
made no LLM calls (no traces) OR every completion came back empty — and labels
it MODEL_ERROR instead of a clean ENV_DONE with reward 0. This is the dead-proxy
/ unreachable-gateway signature (a *correct* rollout is never reclassified).
- Sandbox provisioning: setup-phase failures are tagged so retry-exhaustion
classifies SANDBOX_ERROR (e.g. DaytonaValidationError) instead of generic ERROR,
and the error_type is recorded in metadata.
- Add TerminationReason.MODEL_ERROR (infra family) + mask_model_error (base.yaml
True); map Daytona provisioning errors to SANDBOX_ERROR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(terminus2): self-heal harbor version in the sandbox instead of a bare ready-marker
The install skipped on any `.terminus2-ready` marker, so a snapshot that baked an
old harbor (0.3.0) kept it forever — the version bump only mattered on a manual
rebuild. Skip now only when harbor is ALREADY at the pinned version; otherwise
upgrade in place (create the venv only if missing, don't clobber). A stale 0.3.0
snapshot self-heals to 0.13.2 on next run. Rebuild the snapshot to avoid paying
the reinstall per cold sandbox.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(daytona): deliver long-exec completions — session polling + TCP keepalive
The one-shot process.exec rides a single HTTP request that stays byte-silent
until the command completes; NAT middleboxes on the client path (WSL2
measured: drop at ~245-250s idle) silently discard the flow, so any command
over ~4 minutes finishes in the sandbox but its response is black-holed —
the call blocks until the SDK read timeout (budget+65s) and the episode is
mislabeled TIMEOUT. Bisect: 240s delivered; 250/260/270/300/315/330s all lost.
Two independent defenses:
- execs with budgets >180s (RLLM_DAYTONA_SESSION_EXEC_THRESHOLD_S) ride a
session command: async submit + adaptive short polls (1s→15s) + log fetch;
no connection ever idles long enough to be dropped. In-sandbox coreutils
timeout stays the deterministic killer (exit 124 → SandboxCommandTimeout).
- TCP keepalive (60s probes) injected into the SDK HTTP pools, best-effort:
keeps L4 flow-table entries alive for every other call. Validated: a 300s
one-shot exec delivers with keepalive, is lost 6/6 without.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli-harness): trust the sentinel on exec timeout; classify ExecCompletionLost
A SandboxCommandTimeout no longer unconditionally labels the episode
TIMEOUT: the driver's outcome sentinel is consulted first (it survives
transport loss). Clean sentinel → ENV_DONE with an ExecCompletionLost
audit marker in episode metadata (reward is valid; the transport event
stays visible in tracking); typed sentinel → mapped reason; no sentinel →
TIMEOUT as before. Also adds live regression tests (gated on
DAYTONA_API_KEY / RLLM_LIVE_SLOW=1) asserting exec completions are
delivered past the NAT drop line instead of hanging to the read timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(eval): align no-reward-file test with the typed error taxonomy
Upstream terminal-rl added a test asserting the pre-taxonomy RuntimeError;
this branch replaces that raise with the typed RewardFileNotFoundError
EvalOutput (same intent — a missing verdict is an infra error, not a
score of 0 — now engine-classifiable). Update the test accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(sandbox): fix stale max_concurrent default assertion (4 -> 64)
Upstream raised SandboxedAgentFlow.max_concurrent's default to 64 but the
configure() test still asserted 4 — failing on a clean terminal-rl checkout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff-format daytona backend + delivery test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(integration): skip keepalive delivery test on transient Daytona 5xx
A fast server-side error is a platform blip, not the delivery-loss
regression (loss = hanging to the read timeout, still asserted).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…caps (#722) Per-task [environment] cpus/memory_mb/storage_mb are baked into task.toml at dataset-build time, so an over-provisioned default (e.g. tmax's 4 CPU / 8 GiB, 4x a typical Terminal-Bench task) can only be shrunk by rebuilding. Add three provider-agnostic caps read at sandbox-creation time in _sandbox_resource_kwargs: RLLM_SANDBOX_MAX_CPUS (float) — max physical cores RLLM_SANDBOX_MAX_MEMORY_MB (int) — max memory in MB RLLM_SANDBOX_MAX_STORAGE_MB (int) — max disk in MB (Daytona only) Each clamps the task's declared value down (min) when set (>0): a task already at/below the cap is untouched, and a task declaring nothing stays at the backend default (a cap only lowers, never raises or introduces a value). Modal Sandboxes bill on reserved CPU+memory/sec, so capping cuts the per-rollout bill proportionally without a rebuild. Daytona cpu floors to 1 for fractional caps. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…onors the tunnel daemon (#723) * feat(gateway): whitespace heartbeat keeps slow non-streaming completions alive through tunnel timers Middleboxes on the response path silently kill LLM responses that stay byte-silent while the model generates — measured: Cloudflare quick tunnels 524 at a 120s origin read timer, ngrok 503/ERR_NGROK_3004 at ~300s. Harbor's Terminus-2 calls are non-streaming, so TTFB = full generation time; on terminal-bench-2 with deepseek-v4-pro at reasoning_effort=max (p50 63s, p90 134s, max 502s per call) this killed 28/267 episodes (10.5%) and burned 31.7h of slot time in one eval. ReverseProxy._handle_non_streaming now runs the upstream call as a task: - finishes within heartbeat_initial_delay_s (50s) -> plain response, true status code — fast successes AND typed errors are byte-identical to today - slower -> the response commits as chunked 200 and a single space (legal JSON leading whitespace, invisible to every parser) is emitted every heartbeat_interval_s (25s), so no middlebox ever sees an idle response; the upstream JSON then passes through unchanged - upstream failure after commit -> parseable OpenAI-style error object (gateway_upstream_error); a hung upstream is bounded by heartbeat_budget_s (1h) -> gateway_upstream_timeout — a heartbeat proves the path is alive, it never fakes progress - heartbeat_interval_s <= 0 disables; streaming requests are untouched Validated A/B through a real quick tunnel with a 150s upstream: heartbeat off -> 524 at 125.7s (the production failure); heartbeat on -> 200 at 150.7s with the parsed body identical to the upstream payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eval): honor the rllm tunnel daemon (resolution parity with training) rllm eval hardcoded a fresh Cloudflare quick tunnel per run, ignoring 'rllm tunnel setup'/'rllm tunnel up' entirely — the configured ngrok daemon only ever applied to training. Quick tunnels enforce the 120s read timer above, so slow-model evals kept dying regardless of tunnel config. Eval now resolves like training: $RLLM_GATEWAY_TUNNEL -> running daemon -> quick-tunnel fallback (with a warning naming 'rllm tunnel up'). When the resolved tunnel is a daemon URL, the gateway binds the port the daemon actually forwards to — read from the daemon state file's 'upstream' (new live_tunnel() helper), not from config, so it follows the daemon across port/backend changes. A bare URL with no matching daemon state falls back to the tunnel config's port with a warning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sampling out of FireworksEngine intermittently died on httpx ReadTimeout/ReadError, logged as "Sampling failed permanently after 1 attempts" with an empty message. _completions_with_retry classified transient errors by string-matching str(exc)/class name, but timeouts and transport errors have an empty str(exc) and a class name matching none of the markers, so they were treated as non-transient and gave up on the first attempt (the SDK's own retry can't help: its 300s budget is smaller than the per-attempt http_timeout). Retry fix: - Classify httpx.TimeoutException / TransportError as transient by type, so read timeouts and connection resets (ReadError) are retried instead of failing the whole sampling call. - Include the exception class name in retry / permanent-failure logs, since timeouts render an empty message. Timeout config consolidation (no change to effective defaults): - sample_timeout stays 600 (the prior default). On a streaming SSE call this is httpx's per-read timeout — a cap on inter-token silence, not total generation — so it does not clip long-but-active generations. Made rollout_engine.sample_timeout the single source of truth; fireworks_infra.deployments.rollout.sample_timeout now follows it. - Removed dead training.client_timeout (no consumer in the repo or the training SDK). - Grouped the training-step RPC knobs (step_timeout / step_max_retries / step_retry_backoff_s / weight_sync_timeout) as one timeout + retry policy. - Shared trainer readiness timeout: reference follows policy.timeout_s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…op-health monitoring (#727) * fix(fireworks): bound sampling-retry hold time to curb duplicate churn FireworksEngine is wired as the gateway's in-process handler, so _completions_with_retry runs inside the agent's HTTP call and holds that connection (byte-silent, no heartbeat on the cumulative path) for the whole retry sequence. The prior 10/20/30/40s backoff could hold ~100s+, past the client/tunnel tolerance — the client then re-sends, surfacing as TokenAccumulator "duplicate: regenerating in place" churn plus wasted regeneration (esp. during the every-step weight sync that resets connections). Keep the transient-error classification, but cap the total retry wall-clock (_RETRY_BUDGET_S=90s) and per-retry backoff (_RETRY_BACKOFF_CAP_S=15s): ride out the common transient reset, then fail fast so a persistent outage is released to the client's own retry instead of stalling the connection. Permanent-failure log now distinguishes "retry budget exhausted" from "permanent" and reports elapsed seconds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): take per-request CPU off the single event loop for high concurrency The gateway serves all rollout LLM calls on one asyncio event loop (a thread inside the trainer process). At high concurrency (n_parallel_tasks in the hundreds) that loop can't flush responses / fire heartbeats because it's busy doing per-request CPU work, so connections stall and the agent's retry stack re-sends identical requests -> TokenAccumulator "duplicate" flood + rollout timeouts. Two contained fixes that target the measured hotspots: - Offload tokenizer rendering off the loop: turn-0 render_ids (FireworksEngine.get_model_response) and the cumulative bridge_to_next_turn (proxy handle()) now run via run_in_executor. Tokenizers release the GIL, so these parallelize across worker threads AND the loop stays free to service I/O. turn-0 render on ≤120K-token prompts was the dominant blocker (matches the turn=1-dominant duplicate distribution). - Stop capturing raw_request/raw_response on traces by default (capture_raw_payloads=False). Training reads only token-id/logprob/message fields (rllm.engine.trace_converter), so serializing the full ≤120K-token prompt + response via model_dump was pure per-request CPU on the loop. Does not change training data or trace token/logprob content. Follow-ups (not in this commit): whitespace heartbeat on the cumulative path; run the gateway in its own process to remove GIL contention with the trainer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): extend whitespace heartbeat to the cumulative (turn 1+) path The keep-alive heartbeat only wrapped the turn-0 chat path; cumulative turn-1+ requests were byte-silent for their whole (queued + generating) duration, so a tunnel/NAT idle timer would cut the connection and the agent would re-send — surfacing as TokenAccumulator "duplicate" churn. Extract the heartbeat into a shared `_respond_with_heartbeat(result_coro)` and route both the turn-0 (`_non_streaming_result`) and cumulative (`_cumulative_non_streaming_result`) paths through it. The cumulative result is now produced by a coroutine returning (bytes, status); its accumulator side effects (ingest_turn / update_prefix) run under asyncio.shield, so they still complete exactly once even if the client drops mid-flight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): loop-health monitor to diagnose event-loop starvation Adds a diagnostic-only background task that logs, every ~20s: gateway loop health: lag_ms p50/p99/max | thread_cpu=% | inflight cur/max and stamps `inflight` + `loop_lag_ms` onto each TokenAccumulator duplicate line. - lag = event-loop scheduling overshoot (how long the loop couldn't run). - thread_cpu = the loop thread's own CPU utilisation, which disambiguates why: high lag + high thread_cpu -> self-CPU bound (offload / do less) high lag + low thread_cpu -> loop thread starved of the GIL by the trainer -> a separate gateway process (mode=process) helps - inflight = concurrent in-flight generations (tracked around the heartbeat result task, so it's accurate for buffered and heartbeat-streamed responses). Lets the gateway-in-own-process decision be made from data rather than guessed. No behavioural change. Validated: healthy=~1ms/0%, busy-wait=246ms/89%, idle-block=160ms/24% (the two high-lag signatures are cleanly separable). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): orjson on the hot JSON paths to cut per-core CPU Loop-health data showed the gateway loop thread pegged at ~99% of one core at ~350 concurrent (self-CPU bound, not GIL-starved), with JSON (de)serialization the dominant cost. Add a fastjson helper (orjson with a stdlib fallback + a type-safety net) and swap the per-request hot paths: - middleware: request-body parse + param-injection re-serialize - proxy: request-body parse + non-streaming/cumulative response serialize - token_accumulator: per-message fingerprint (hash orjson bytes directly) Measured on representative payloads: response serialize (16K token_ids + logprobs) 20.1ms -> 1.8ms (11.2x), request parse (~480KB) 2.7x, fingerprint 6x. Round-trip identical; sorted-key fingerprint stable and internally consistent. This raises single-core request capacity (may be enough on its own); the multi-process split remains the path to using all cores if not. Streaming/error/worker JSON paths (cold for the Fireworks non-streaming setup) left on stdlib. orjson added as a declared dependency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): orjson httpx bodies + offload trace build off the loop Two more single-process loop-unblockers, from the loop-health finding that the gateway thread is pegged at ~98% of one core with multi-second lag: 1) httpx orjson patch (fireworks_engine): the Fireworks SDK POSTs the full ≤120K-token prompt via client.post(json=payload); httpx serialized it with stdlib json on the event-loop thread (~4ms+/req, the dominant on-loop cost we couldn't reach from our own code). Patch httpx._content.encode_json to use orjson (compact/UTF-8/no-NaN — matches httpx's wire semantics). Guarded, best-effort: probes the (headers, ByteStream) shape against the real function and skips (keeping stdlib) on mismatch, so an httpx upgrade can't break sampling. Measured ~4ms -> sub-ms on a 120K-int prompt. 2) Trace build offloaded (proxy): build_trace_record + model_dump (copying the ≤120K/16K token-id lists) ran inline on the loop before the response returned. Move them into the executor via _persist_trace/_build_trace_data and off the response critical path; only the async store write touches the loop. sync_traces still forces synchronous completion. Both keep the loop responsive (fewer heartbeat misses -> fewer connection cuts / duplicates). They do not lift the GIL 1-core ceiling — multi-process remains the path to using all cores. 236 gateway tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(gateway): orjson for the Fireworks SDK streaming response parse async_completions_stream parses every SSE chunk with stdlib json.loads on the event-loop thread; across many concurrent streams (one chunk per token) that per-chunk parse is a continuous on-loop cost. Swap the SDK sampling module's `json` reference for a shim whose `loads` is orjson and that delegates all other attributes (dumps, JSONDecodeError, …) to stdlib — so nothing else changes. orjson.JSONDecodeError subclasses ValueError, so the SDK's `except (ValueError, TypeError)` still catches malformed chunks. Measured ~3x per chunk (1.9µs -> 0.6µs); this is eliminated work (orjson does the same parse in less CPU), so unlike the trace offload it raises the throughput ceiling. Total impact scales with chunks-per-response (streaming granularity) — the next loop-health thread_cpu reading will show the real drop. Best-effort/idempotent install next to the existing httpx + header patches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): run the gateway in its own process (num_workers=1) The gateway runs as a thread inside the trainer, sharing one GIL; loop-health showed the gateway loop pegged at ~98% of a core, starving the trainer's Python (weight-sync, episode processing). This adds an opt-in separate-process mode so the gateway gets its own event loop / GIL. The only coupling is that the in-process handler (FireworksEngine) can't cross a process boundary, so the subprocess rebuilds an equivalent one from a serializable recipe — kept backend-agnostic in the gateway package: - FireworksEngine.handler_factory_spec() -> (import_path, config): attaches to the SAME deployment via the sampler's base_url/model (no re-provision); API key from env, never serialized. (renderer_family now retained for this.) - rllm/gateway/worker_handlers.py::build_fireworks_handler(config): rebuilds tokenizer + DeploymentSampler + FireworksEngine + create_tinker_handler. - gateway package: `python -m rllm_model_gateway --handler-factory module:func --handler-config x.json` dynamic-imports the factory and passes the built local_handler to create_app. Package never names a backend. - GatewayManager: rllm.gateway.num_workers (default 0 = trainer thread, unchanged). 1 = single separate process via _start_gateway_subprocess (reuses _start_process spawn/health-poll; longer startup budget for engine build; temp config cleaned up in stop()). >1 raises NotImplementedError (needs the session-sticky front — the next step). Trainer<->gateway comms (weight_version, traces) already go over HTTP, so they work unchanged against a subprocess. Validated: generic factory plumbing end-to-end (dummy factory served a request); spec extraction + spec<->factory key contract; 236 gateway tests pass. The live Fireworks build (real DeploymentSampler attach) is validated on a training run. Expected: frees the trainer from GIL contention; gives the gateway a dedicated core (~98%->~100%, marginal). Does NOT by itself clear high concurrency on the gateway — one core is still one core; that needs num_workers>1 (front router). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): silence per-request uvicorn access logs in subprocess mode The separate-process gateway ran uvicorn at info, so every "POST /v1/chat/completions 200 OK" was logged — a flood at hundreds of concurrent requests. Disable uvicorn access_log unless log_level=debug (thread mode already suppressed these by running uvicorn at 'warning'). Our own INFO logs (loop-health, TokenAccumulator duplicates) are on the rllm_model_gateway logger and are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(trainer): event-loop health monitor for the trainer loop Mirrors the gateway's loop-health monitor so the trainer's single event loop is observable too — to catch when the *trainer's* data pipeline (on-loop enrich_episode_with_traces, advantage/batch prep) becomes the bottleneck now that the gateway can run in its own process. - rllm/utils/loop_health.py: reusable run_loop_health_monitor(label, gauges=...) logging lag_ms p50/p99/max + thread_cpu% (+ optional gauges) every ~20s. Same discriminator as the gateway: high lag + high thread_cpu = self-CPU bound; high lag + low thread_cpu = GIL-starved. Logger pinned to INFO so it stays visible when the rllm.* hierarchy is raised to WARNING. - AgentFlowEngine: store n_parallel_tasks; add read-only `inflight`/`pending` gauges (derived from the concurrency semaphore) for the monitor. - unified_trainer._fit_fully_async: start the monitor as a background task (label "trainer", gauges=inflight/pending), cancel on shutdown. Diagnostic only. Validated: lag/thread_cpu discriminator + gauge rendering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-sync-window quota in SyncCoordinator with a continuously
evaluated capacity, mirroring AReaL's StalenessManager.get_capacity:
dispatch allowed iff in_flight_groups < (1+staleness)*mini_batch (staleness)
and running_rollouts < n_parallel_tasks (concurrency)
Budget frees on every consume, filter, and rollout-task completion via
_signal_capacity() -- not only at weight sync -- so freed slots refill
immediately instead of the old front-load-then-idle burst within a step.
on_sync_complete now only bumps the version.
New metrics: async/running_rollouts, async/max_in_flight_groups,
async/max_concurrent_rollouts. Adds tests/trainer/test_sync_coordinator.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ity summarization (#725) * feat(harness): terminus2-native — Terminus-2 over native tool-calling with harbor-parity summarization Same terminal interaction model as terminus2 (tmux keystroke batches + screen observations), but the conversation rides native tool-calling rails: execute_commands tool calls, role:tool results, and reasoning_content echoed verbatim on assistant messages so think-mode models (DeepSeek-V4-Pro) retain reasoning across turns. Measured on Fireworks: the user-message path renders echoed reasoning to zero prompt tokens; the tool-calling path renders it fully. Context management is ported verbatim from harbor terminus_2.py so eval deltas stay attributable to the transport alone: - three-subagent summarization dance (summary -> fresh-context questions -> answers), prompt texts byte-identical to harbor's - reactive fallback chain on context-length errors: unwind whole turns from the tail -> full dance -> short summary -> no-LLM handoff -> "Technical difficulties" fabricated reply (episode never crashes) - proactive summarization when estimated free tokens < 8000; tokens estimated from real usage anchors + chars/4 instead of litellm's model map (whose unmapped-model 1M default silently disabled summarization in raw harbor) - context-length errors fast-fail out of the retry loop (a doomed payload retried 4x is just 4 identical failures) Knobs: enable_summarize, proactive_summarization_threshold, context_limit_tokens (env RLLM_TERMINUS_CONTEXT_LIMIT wins so eval runs can pin the model's real limit, e.g. 240000 for v4 on Fireworks). Validated end-to-end on TB2 (v4-pro, effort=max): the dance fired in production and the 207-step episode recovered to a PASS; 100% reasoning persistence across all episodes; blended pass@1 ~66% / pass@3 ~76% vs 54.3 for the classic user-message harness (official in-house: 67.9). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: apply ruff format Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(data): tb-v2 dataset catalog entries + HF/local task builders Adds tb-v2-debug / tb-v2-opus-pass-at-1 / tb-v2-full (and friends) to the dataset registry, backed by new hf_tasks_builder/local_tasks_builder that materialize task archives from HF (mobius-lab/tb-v2-tasks) or local dirs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(gateway): preflight the gateway port before expensive setup The gateway binds 0.0.0.0:<port> from a background thread/subprocess, so a bind conflict surfaced as an opaque 30s health-poll timeout — after minutes of Fireworks infra provisioning. Probe-bind the port at trainer init (before the backend provisions anything) and again in start(), and raise GatewayPortInUseError naming the holder-lookup command and, when the port is the tunnel daemon's upstream, the likely concurrent-run cause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(fireworks): retry raw ssl.SSLError during sampling The Fireworks SDK's SSE stream can surface raw ssl.SSLError (e.g. bad_record_mac) unwrapped by httpx; the transient classifier missed it, failing the sampling call permanently on attempt 1 — 26% of episodes died to this in a single run. Classify ssl.SSLError as a network error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(terminal-rl): qwen3p5-35b debug launch script + cookbook updates Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): pin fireworks-training-cookbook to the locked commit The unpinned git ref resolved to cookbook HEAD, which now requires fireworks-ai>=1.2.0a86 and conflicts with our ==1.2.0a83 pin, making rllm[fireworks] unresolvable on fresh installs. Pin the commit recorded in uv.lock (the combo running in production). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(lint): apply pre-commit ruff fixes to PR-changed files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sandbox): demote per-command failure logs to debug A non-zero verifier exit is the expected outcome for an unsolved task; the raised exception already carries the detail and callers pick the severity. Matches the docker backend, which already logged at debug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(terminal-rl): tune qwen3p5 debug config (context 101k, port 9091, 4 replicas) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: signalrush <269811712+signalrush@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local>
…rbor-par…" This reverts commit b5eeb93.
…m_workers>1) (#728) * feat(gateway): ConsistentHashPolicy for multi-worker session sharding Deterministic session_id -> worker mapping (stable sha256 over the fixed registered worker set, ordered by url) for num_workers>1: a session's chat turns AND its trace/session-control ops must always land on the same worker (which holds that session's in-memory accumulator/traces/params). Health churn doesn't remap sessions (a dead worker's sessions fail cleanly rather than silently landing on a worker that lacks their state). Sessionless requests fall back to least-loaded. Phase 1 of num_workers>1; not yet wired into a front. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): thin session-sharding front for multi-worker mode Adds a dedicated thin reverse proxy (rllm_model_gateway.front) that sits in front of N backend gateway workers and routes by session_id: - /sessions/{sid}/... -> the owning worker (ConsistentHashPolicy), forwarding the ORIGINAL path so the worker's own middleware keys its per-session state; responses are streamed (SSE + heartbeat pass through). - POST /sessions (create): sid from the body -> owner. - global ops (weight_version / flush / reload / batch_delete) -> fan out to all workers, aggregating the result. - /health. Run via `python -m rllm_model_gateway --front --worker URL --worker URL ...`. The front never parses chat bodies or holds session state — workers (ordinary num_workers=1 gateways) do all of that; the front stays thin so one core routes for many workers. Validated end-to-end with 2 dummy workers: sticky routing (same sid->same worker), balanced distribution, traces routed to owner, weight_version fan-out reaching both workers, delete routed to owner. Not yet spawned by GatewayManager (next commit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): GatewayManager spawns N workers + front for num_workers>1 Generalizes the separate-process gateway to multiple workers: - num_workers==1: one gateway on port (unchanged). - num_workers>1: N worker gateways on port+1..port+N (each a full local_handler gateway via the handler-factory, cumulative ON), behind a thin `--front` session-sharding proxy on `port` (the tunnel target). Each worker is health-polled with extra startup headroom, then the front. Refactors spawning into _gateway_cmd (worker vs front) + _poll_health(port); tracks all processes in _processes and terminates them (front first) in stop(). Trainer<->gateway control plane is unchanged: it hits the front (port), which routes/fans-out — so set_weight_version / aget_traces / adelete_session need no changes. Validated via a manager-driven smoke (2 workers + front): requests shard across both worker processes, sticky routing, weight_version fan-out reaches both, 3 processes spawned. Drops the num_workers>1 guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gateway): tag per-worker logs with the worker port (num_workers>1) With multiple gateway workers behind the front, their `loop health` and `TokenAccumulator duplicate` INFO lines interleave indistinguishably. Prefix both with a worker label (the worker's port) so a hot/among-N worker is identifiable. Empty label for the single-process case (no change there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): silence httpx per-request INFO logs in gateway processes httpx logs one INFO line per request; with the multi-worker front forwarding every chat call to a worker (and workers calling Fireworks), that floods the trainer stdout ("INFO:httpx:HTTP Request: POST .../v1/chat/completions 200 OK"). Set the httpx/httpcore loggers to WARNING in the gateway processes unless log_level=debug. Our own INFO logs (loop-health, duplicates) are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(gateway): ruff import-sort + format Fixes pre-commit: I001 unsorted-imports in manager.py and ruff-format line collapsing in proxy.py/server.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The async continuous-dispatch generation loop drives rollouts via process_task_with_retry, bypassing AgentFlowEngine.run() and its end-of-step batch_delete. Every completed rollout's gateway session + traces (fat cumulative token-id lists) therefore linger in the gateway's in-memory store for the whole run, growing the per-worker heap unbounded. CPython's cyclic GC then walks that graph on the event-loop thread, stalling request handling (loop-health lag spikes, thread_cpu ~100%, /health timeouts). - unified_trainer._generation_loop: delete each gateway session in a try/finally once the episode is buffered (its traces are already fetched into it), bounding the store to the in-flight set. No-op for gateway-less engines; also frees the session on permanent failure. - gateway server.main(): gc.freeze() the static tokenizer/renderer/sampler and raise the gen-2 threshold. Scoped to the standalone gateway process (main() never runs for the in-trainer thread-mode gateway) so it can't perturb the trainer's GC. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gateway binds 0.0.0.0:<port> from a background thread/subprocess, so a bind conflict surfaced as an opaque 30s health-poll timeout — after minutes of Fireworks infra provisioning. Probe-bind the port at trainer init (before the backend provisions anything) and again in start(), and raise GatewayPortInUseError naming the holder-lookup command and, when the port is the tunnel daemon's upstream, the likely concurrent-run cause. Riders: typing->collections.abc import lint in worker_handlers.py and loop_health.py (zero behavior change). Co-authored-by: Tianhao Wu <thw@gpu-dp-6wgmd-rj5sb.slurm-compute.slurm.svc.cluster.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Harbor's Terminus-2 enables context summarization ("compaction") by
default: when the context fills it spawns summarization subagents that
compress history, which fragments the trajectory the gateway captures
for training.
Plumb an `enable_summarize` knob through the terminus2 harness
(RLLM_TERMINUS_ENABLE_SUMMARIZE -> in-sandbox driver -> Terminus2) and
have the terminal-rl cookbook read TERMINUS_ENABLE_SUMMARIZE. It defaults
to on when train.py is run standalone (Harbor's default); the train_*.sh
scripts set it to 0 to turn compaction off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s; add RLLM_HARNESS_VERIFIER_TIMEOUT_S cap (#736) * fix(sandbox): size Modal/Daytona lifetime from effective agent + verifier timeout Two problems made rollouts (esp. SWE-bench Verified) look like they "run forever" or get reaped mid-verify: 1. The agent timeout used for sizing was `task.metadata["agent_timeout"] or RLLM_HARNESS_RUN_TIMEOUT_S`, so a task's large baked-in agent_timeout bypassed the operator cap. Now RLLM_HARNESS_RUN_TIMEOUT_S is a hard cap on the per-task value (matching cli_harness._effective_timeout). 2. The lifetime added install + a flat 600 slack but not the (enforced) verifier_timeout in a way that reliably covers a long verifier — a task whose verifier legitimately runs ~3000s (verifier.timeout_sec) outlived the sandbox. Now: verifier declared -> lifetime = agent + verifier_timeout + 300 verifier absent -> lifetime = agent + 600 raised to the RLLM_SANDBOX_TIMEOUT_S override. So the sandbox is always slightly longer than agent + the enforced verifier budget, with no config needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(sandbox): RLLM_HARNESS_VERIFIER_TIMEOUT_S — operator cap on the verifier timeout Symmetric to RLLM_HARNESS_RUN_TIMEOUT_S (agent). Tasks bake in a per-task verifier.timeout_sec (SWE-bench Verified declares 3000); this env caps it as a hard ceiling, applied to both the enforced verifier timeout (ShellScriptEvaluator) and the sandbox-lifetime sizing via a shared _effective_verifier_timeout(task). Unset = use the task's declared value. Lets an operator shrink an over-declared verifier for fast iteration without editing task.toml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…reworks (DPPO, CISPO, GSPO) (#713) * feat(algorithm): unified custom-loss abstraction across verl/tinker/fireworks (DPPO) A single per-token loss-term contract — `LossContext -> (per_token_loss, agg_mask, metrics)` — that runs identically in-process on verl and via `forward_backward_custom` on tinker/fireworks (the two managed backends share that mechanism). Subsumes the old policy-vs-aux split: ECHO is now just a term over the observation mask. Core (rllm/trainer/algorithms/loss.py): - @register_loss registry, LossContext, resolve_loss_terms, loss_plugins import hook. - Built-in terms: dppo_tv, dppo_kl, ppo_clip, env_prediction (ECHO). dppo_tv is unit-verified to reproduce verl 0.8's compute_policy_loss_dppo_tv per-token loss. Backends: - verl: register rLLM terms into POLICY_LOSS_REGISTRY on Ray workers (patch hook); never shadows native dppo_tv/dppo_kl. ECHO stays on CustomPPOLoss._apply_aux_losses. - tinker/fireworks: shared forward_backward_custom adapter folds surrogate + ECHO into one pass (no separate aux CE pass); fireworks forces GradAccNormalization.NONE. Public API: @rllm.register_loss / rllm.LossContext (same lazy-export style as @rllm.rollout / @rllm.evaluator). Config: algorithm.losses / loss_plugins / mu_source (mu defaults to inference log-probs — the tmax DPPO choice). Back-compat: native loss_fn (vanilla/ppo/grpo) and aux_losses/env_loss_coef paths are unchanged; the unified path engages only for rLLM-registered losses or an explicit algorithm.losses list. Tests: tests/unified_trainer/test_custom_loss.py (12). See design/unified-custom-loss.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): deprecate the standalone aux-loss framework; ECHO is now a single loss term Collapse the two parallel systems into one. ECHO is defined once as the `env_prediction` term (`@register_loss("env_prediction", aux_mask=MASK_OBSERVATION)`); the old `aux_loss` framework (`@register_aux_loss`, `AuxiliaryLoss`, `EnvPredictionLoss`) becomes a thin deprecation shim that emits `DeprecationWarning` and delegates to the unified registry. - loss.py: `register_loss(name, aux_mask=...)` records a static token region used by the managed CE realization; `resolve_additive_terms()` resolves `aux_losses`/`env_loss_coef` into unified `ResolvedTerm`s (with `.mask`). ECHO = `env_prediction` term. - verl: `_apply_aux_losses` now evaluates the term function over a LossContext (single source of truth) instead of hardcoded CE. - tinker/fireworks: aux executors source positions/coef from `ResolvedTerm` (efficient CE realization unchanged); `build_aux_losses` -> `resolve_additive_terms`. - aux_loss.py: deprecation shim; `build_aux_losses` is a non-warning internal alias, `register_aux_loss`/`AuxiliaryLoss`/`get_aux_loss` warn. `EnvPredictionLoss` removed. Back-compat: `aux_losses` / `env_loss_coef` / `adv_estimator=echo` still work (resolve to the `env_prediction` term). Migration: `@register_aux_loss` -> `@rllm.register_loss(name, aux_mask=MASK_OBSERVATION)`, `aux_losses:` -> `losses:`. Tests: test_aux_loss.py rewritten for the deprecation/delegation (33 unified-trainer tests pass). See design/unified-custom-loss.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): verl-style single-loss selector; remove the auxiliary-loss framework Per review: drop the `losses` list and the standalone aux-loss system; mirror verl's model of one loss selected by a string. A loss is a single function that returns the COMPLETE scalar objective and aggregates via a backend-injected `ctx.aggregate` — exactly like a verl POLICY_LOSS_REGISTRY function. - loss.py: LossContext gains `aggregate`; losses return `(scalar, metrics)`. `resolve_loss` returns the single selected loss (or None for a backend-native one). Built-ins: ppo_clip, dppo_tv, dppo_kl, ppo_clip_env. dppo_tv still matches verl 0.8's kernel (unit-tested). - ECHO is no longer a framework: it's the `ppo_clip_env` loss, which adds the obs-token CE term inside its own body. `adv_estimator=echo` defaults loss_fn=ppo_clip_env (env_loss_coef 0.05). `algorithm.loss_params` carries loss-specific params (verl-style). - verl: CustomPPOLoss runs the rLLM loss in-process over a LossContext (aggregate=agg_loss with global-batch norm), native kernel otherwise. Removed _apply_aux_losses and the POLICY_LOSS_REGISTRY shim/worker hook. - tinker/fireworks: forward_backward_custom runs the single loss; removed the separate aux CE passes (_get_aux_loss_futures, _build_aux_datums) and aux metric plumbing. Deleted: aux_loss.py, tinker/aux_loss.py, verl/custom_loss.py, design/auxiliary-losses.md, test_aux_loss.py; config `losses`/`aux_losses`. Tests rewritten (19 pass). See design/unified-custom-loss.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): add CISPO and GPG built-in losses CISPO (arXiv:2506.13585): clips the importance-sampling weight with a stop-gradient but keeps every token's gradient through log_prob (no token dropped, unlike PPO clip). GPG: clip-free REINFORCE-style policy gradient with group-normalized advantages. Both are per-token functions of (pi, mu, advantages, mask) — direct fits for the LossContext, so they run on all three backends. CISPO is unit-verified against verl 0.8's kernel and against ppo_clip's gradient behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): add GSPO + LossContext.seq_reduce for sequence-level losses GSPO (arXiv:2507.18071): a sequence-level length-normalized importance ratio, clipped PPO-style and applied per token, using verl's stop-gradient identity so the value is the detached sequence ratio while the gradient flows per-token through log_prob. Adds one primitive to the abstraction: `LossContext.seq_reduce(values, mask, reduction)` reduces over each sequence and broadcasts back to per-token — a row on verl (B,T), the whole datum on the managed path. `ctx.aggregate` gains a `mode` override so GSPO forces seq-mean-token-mean. gspo is unit-verified against verl 0.8's kernel; seq_reduce and per-token gradient flow are tested. This is all `geo_mean` (GMPO) needs too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): layered loss resolution — entry-point discovery + loss_plugins on workers get_loss/is_custom_loss now fall back to lazy `rllm.losses` entry-point discovery on a registry miss (mirrors rllm.eval.agent_loader/evaluator_loader), so a pip-installed loss package is found with no config and works across process boundaries (incl. verl Ray workers, which call get_loss there). CustomPPOLoss also imports algorithm.loss_plugins on the worker so the explicit-list path registers there too. Explicit inline import still works unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): native-first loss routing — prefer a backend's fused kernel over the custom path resolve_loss now takes the backend's native fused-kernel menu and returns None when loss_fn is in it, so the fast native kernel is used instead of the rLLM custom path — even when an rLLM loss of the same name exists. Only losses a backend can't run natively fall back: - verl: CustomPPOLoss runs the in-process rLLM loss only if loss_mode is NOT in verl's POLICY_LOSS_REGISTRY (so native dppo_tv/gspo/cispo on 0.8 win). - tinker: native_losses=TINKER_KNOWN_LOSSES → cispo/ppo/dro use the server string path. - fireworks: native_losses=BUILTIN_LOSSES (grpo/importance_sampling/dapo/dro/gspo/cispo) use the builtin server kernel; only dppo_tv/ppo_clip_env take forward_backward_custom. Avoids the ~3x forward_backward_custom cost whenever a fused kernel exists. Losses routed native take hyperparameters from backend config (eps_clip→clip_ratio), not loss_params. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(algorithm): centralize native-loss detection in native_loss_names(backend) One map of what each backend has a native fused kernel for, derived from the backend's own source of truth (no drift across versions): - verl: POLICY_LOSS_REGISTRY (vanilla, dppo_tv, dppo_kl, gspo, sapo, gpg, clip_cov, kl_cov, geo_mean, cispo, bypass_mode) - tinker: tinker.types.LossFnType (cross_entropy, importance_sampling, ppo, cispo, dro) - fireworks: builtin_losses.BUILTIN_LOSSES (grpo, importance_sampling, dapo, dro, gspo, cispo) Replaces the scattered inline derivations (verl POLICY_LOSS_REGISTRY check, tinker TINKER_KNOWN_LOSSES, fireworks _native_loss_names) with native_loss_names("<backend>"); all three now feed resolve_loss's native-first routing from this one place. Returns ∅ when the backend isn't importable → falls back to the rLLM custom path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): rename ppo_clip_env loss → echo Shorter, matches adv_estimator=echo (which selects it). No behavior change — echo is rLLM-only (no backend has a native kernel), so it always runs the custom path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(algorithm): rename gpg loss → reinforce Our gpg loss is exactly -advantages*logπ — the REINFORCE loss (no ratio, no clip); the "GPG" name really refers to the advantage normalization, which lives in the estimator, not this loss. `reinforce` is the accurate, recognizable name (verl calls the same loss `gpg`, Fireworks `reinforce`). verl's native-menu references stay `gpg` (that's verl's name). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): skip builtin-loss validation for custom (forward_backward_custom) losses _init_fireworks_infra and resolve_builtin_loss validated algorithm.loss_fn against Fireworks' builtin kernel menu unconditionally at startup, which rejected rLLM custom losses (e.g. dppo_tv) with "Unsupported policy_loss" before they ever reached the client-side forward_backward_custom path. Guard both with resolve_loss(..., native_losses= native_loss_names("fireworks")): a loss that isn't a Fireworks builtin skips the builtin validation/resolution (and self._builtin_loss stays unused, since the custom path returns early). Native/default losses (grpo, cispo, ...) validate as before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(loss): unify loss_agg_mode across backends; fix managed grad-accum normalization The managed (tinker/fireworks) custom-loss adapter hardcoded one aggregation (token-mean-within-datum, cross-datum mean) and ignored algorithm.loss_agg_mode, while forcing GradAccNormalization.NONE on Fireworks. Under gradient accumulation (mini_batch_size/fwd_bwd_group_size > 1) the per-pass means are SUMMED, not averaged, so the gradient — and the effective LR — is inflated by num_fwd_bwd_passes (16x for the a35b DPPO cookbook). verl was already correct: its injected aggregate = agg_loss(..., batch_num_tokens, global_batch_size, dp_size) folds the GLOBAL normalizer into every micro-batch. Unify on one loss_agg_mode (verl's names/semantics), realized per backend: - loss.py: register_loss(agg_mode=) + RLLM_LOSS_AGG_MODE + ResolvedLoss.agg_mode; resolve_loss picks loss-pin > config > DEFAULT_LOSS_AGG_MODE ("token-mean", matching verl + Fireworks' RL guidance). gspo pins seq-mean-token-mean. - custom_loss.py: build_custom_loss is mode-aware (within-seq token-mean for seq-mean-token-mean, else masked sum; cross-seq sum) and adds server_normalized: * Fireworks (True): return a RAW SUM; server divides across the whole window. * Tinker (False): divide client-side by the pass's global count (correct for its single-pass usage; per-pass under accumulation — documented). - fireworks_policy_trainer.py: custom path builds server_normalized=True and optim_step sets GradAccNormalization from resolved.agg_mode (NUM_LOSS_TOKENS for token-mean, NUM_SEQUENCES for seq-mean-*) instead of NONE — accumulation-invariant. Also restore offpolicy/* diagnostics on the Fireworks custom path (proximal-vs- inference log-probs): the custom branch returned before the builtin path computed them, so dppo_tv runs logged no offpolicy metrics (and DPPO's keep-mask depends on exactly that train/inference gap). Tests: agg_mode resolution + pin, register_loss validation, Fireworks accumulation-invariance (raw sums compose), Tinker client-side per-mode divisor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(fireworks): drop diagnostic proximal forward on the custom-loss path The offpolicy/* metrics I restored required a proximal forward pass (_compute_proximal_logprobs) on every fwd-bwd pass — mini_batch_size/ fwd_bwd_group_size extra forwards per optimizer step (16x for the a35b DPPO cookbook), a noticeable training-loop slowdown. On the custom (DPPO) path mu = the inference log-probs already on the datums, so pi_old is never recomputed and that forward is purely diagnostic. Remove it; the loss still emits its own diagnostics (e.g. dppo_tv/mask_frac). The builtin GRPO path is unchanged — it needs proximal for its ratio denominator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gateway): stamp traces with the proxy's fanned-out weight_version (fix async staleness) async/staleness_* read a flat value equal to weight_version (min=mean=max, e.g. 20 at step 21) — every trajectory group was tagged weight_version=0. Root cause: build_trace_record preferred the engine-stamped weight_version in the response body over the proxy's tracked value. In a multi-worker gateway (num_workers>1) each worker subprocess rebuilds its own rollout engine whose self.weight_version is never updated — only proxy.weight_version is (the front fans /admin/weight_version out to every worker). So the engine stamped a stale 0 into every response, clobbering the correct proxy value → all steps 0 → group 0 (_min_weight_version) → staleness = coordinator.weight_version. Fix: prefer the proxy's tracked version (correct on every worker); fall back to the engine-stamped response value only when the proxy isn't tracking one. This matches build_trace_record_from_chunks (streaming), which already used the proxy value only, so the two paths are now consistent. Fixes staleness for the multi-worker subprocess, single subprocess, and in-thread gateway configs. Off-policy metrics were already correct (they compare recomputed proximal vs inference log-probs, not the trace tag), so this is purely the staleness gauge. tests: TestBuildTraceRecordWeightVersion — proxy wins over stale response, base version 0 not clobbered, response fallback preserved when proxy untracked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(gateway): trim the weight_version precedence comment Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * change(algorithm): default loss_agg_mode to seq-mean-token-mean Was token-mean. seq-mean-token-mean weights every sequence (trajectory) equally regardless of length — the GRPO-original behavior and a better default for the variable-length agentic trajectories rLLM targets. Single source of truth is rllm.algorithm.loss_agg_mode (base.yaml). Via the sync_config shared-key mapping it also propagates to verl's actor.loss_agg_mode: a non-null rllm yaml default now wins over verl's token-mean default (previously null let verl's default stand). So the new default applies uniformly to the managed (tinker/fireworks, native + custom-loss) and verl backends. Setting loss_agg_mode=null explicitly still defers to verl's own default. - base.yaml: loss_agg_mode null -> seq-mean-token-mean - AlgorithmConfig: field default + from_config fallback -> seq-mean-token-mean - loss.py DEFAULT_LOSS_AGG_MODE (custom-loss fallback) -> seq-mean-token-mean - test updated Behavior change: runs that didn't set loss_agg_mode and relied on token-mean now get seq-mean-token-mean; pass loss_agg_mode=token-mean to keep the old behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(async): skip fwd/bwd + optim on an all-dropped batch instead of crashing If every trajectory in a batch is dropped as malformed (empty output_logprobs — e.g. failed/overloaded generations that returned no completion), the managed backends called forward_backward([]) and crashed with "ValueError: No data provided to forward_backward", killing the whole run (observed after ~100 steps on a35b when an inference-overload spike made a full fwd-bwd chunk's rollouts all fail). Two guards: - forward_backward_from_trajectory_groups (fireworks + tinker): when transform yields no datums, log a warning and return early (skip forward_backward) with train/num_sequences=0, train/dropped_all_sequences=1. - _training_loop: track trainable sequences across passes; run update_policy + weight sync only when >0. An all-dropped step accumulated no gradient, so stepping the optimizer would apply an undefined/zero-gradient update and bump the LR schedule + trigger a weight sync for nothing. on_training_step_complete is likewise gated so a skipped batch doesn't count toward the sync trigger. Adds async/trained_this_step metric. The transient overload that produced the empty batch is expected; the trainer now rides it out instead of dying. (Root cause of the empty batches — rollout concurrency vs inference replica capacity — is a config concern, separate.) tests: TestAllMalformedBatch — all-empty-logprobs batch -> empty datums + dropped_malformed_sequences count; mixed batch keeps the valid trajectory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fireworks): default bypass_mode=True so the GRPO clip isn't inert in async The managed async loop runs one optimizer step per batch (no inner PPO epochs), so a recomputed proximal equals the current policy (pi_old == pi_theta): the clip ratio pi_theta/pi_old is ~1, the clip does nothing, and the proximal forward is wasted. fireworks_backend already ASSUMED bypass_mode defaults True (its validation reads rc.get("bypass_mode", True)), but the gradient path read the AlgorithmConfig field whose default is None -> False, so it silently recomputed proximal and ran an inert clip. Honor True by default in the policy trainer: pi_old := rollout logprobs, making the clip a real pi_theta/pi_rollout trust region against the actual behavior policy (per-token, correct weight version, from the recorded rollout logs) -- the same 2-policy anchor DPPO/SAO use. Also skips the redundant proximal forward. The offpolicy/* diagnostic (real proximal vs rollout) is only computed when bypass=false, since under bypass prox==rollout makes it trivially 1. Scope: Fireworks only. Tinker never recomputes proximal (already uses rollout logprobs directly), so it's unaffected. verl keeps its own default -- verl's bypass=True path requires rollout_log_probs in the batch and asserts otherwise, so a global default flip would be unsafe there; verl users opt in explicitly. Set rollout_correction.bypass_mode=false to restore the 3-policy proximal path (only meaningful with multi-update-per-batch PPO). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: fix ruff (pre-commit) on the custom-loss files Pre-commit (ruff v0.11.4 lint + format) was failing on the feature's files. Auto-fixed: UP037 (drop quotes from annotations under `from __future__ import annotations`), UP035 (Callable from collections.abc), UP007 (X|Y), F401 (unused torch import in seq_reduce), I001 (import sorting). Manual: wrapped an E501 line in loss.py; `# noqa: E402` on the two imports after pytest.importorskip in test_custom_loss.py (skip-if-no-torch pattern). ruff-format applied to loss.py / unified_trainer.py / verl_backend.py. No behavior change; 27 custom-loss/transform tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(dppo): C=inf (drop truncated-IS) to match the DPPO paper dppo_tv/dppo_kl multiplied the score by a *truncated* importance ratio (min(pi/mu, 20)) via the shared _truncated_is helper. But the DPPO paper (arXiv:2602.04879, Eq. 23) sets C=inf for DPPO: the divergence mask IS the trust region, and Sec. 5.4 ("Pitfalls of Truncated Importance Sampling") shows truncation reintroduces the low-prob-token bias DPPO exists to remove. Use the untruncated ratio as a detached weight on the score; _ratio's +/-20 log-ratio clamp stays purely as inf protection. Remove the now-unused _truncated_is helper (cispo/ppo_clip keep their own finite clip on purpose). test_dppo_tv_matches_verl_formula: reference switched to C=inf and a large-ratio (~150) kept token added so a truncation regression is actually caught (was inert before: random inputs never exceeded the cap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(loss): rename LossContext fields pi/mu/ref -> logp_curr/logp_old/logp_ref `mu` was overloaded — it means the proximal old-policy in one mode and the rollout/behavior policy in another (they only coincide under bypass_mode). Name the fields by role, aligned to verl's own field names, so the distinction is explicit and the bypass identity reads as a statement ("in bypass, logp_old == logp_rollout") rather than a hidden collapse: pi -> logp_curr current policy being optimized (grad) [verl log_prob] mu -> logp_old the ratio denominator (proximal old policy) [verl old_log_probs] ref -> logp_ref KL reference policy [verl ref_log_prob] Pure mechanical rename, no behavior change. Updated the dataclass, all loss bodies + docstrings, the two LossContext construction sites (verl_backend, tinker/custom_loss), and the tests. verl's "ref" mesh / ref_policy_wg is untouched (only the 3 LossContext kwargs changed). 25/25 custom-loss tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(loss): trim overly verbose comments (dppo C=inf, fireworks bypass/offpolicy/all-dropped) Comment-only; no behavior change. Also aligns the fireworks comments with the logp_old naming. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add "Custom loss functions" page (built-ins + how to register your own) New docs/training/custom-loss.mdx under Advanced algorithms: the built-in policy losses (ppo_clip, dppo_tv/kl, cispo, gspo, reinforce, echo), the LossContext contract + aggregate/seq_reduce reducers, loss_agg_mode, native-first backend routing, and a worked custom-loss example (IcePop) with loss_plugins wiring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(loss): expose logp_rollout on LossContext (raw inference log-probs) Adds an optional `logp_rollout` field = the behavior policy's recorded sampling log-probs, distinct from `logp_old` (the ratio denominator). They coincide under bypass_mode but differ when a proximal old-policy is recomputed (non-bypass / verl old_log_probs vs rollout_log_probs). Lets a custom loss reference the true sampling distribution independently of the denominator — needed for train/inference-mismatch corrections (TIS/IcePop), which land in a future PR. Populated on every backend: verl `rollout_log_probs` (when present), managed path always the datum's raw `logprobs` (regardless of any `mu_arrays` proximal override). Documented in the custom-loss page; test asserts it's populated and distinct from logp_old under a proximal override. 26/26 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(config): data-driven estimator→loss default; make echo a normal loss; drop mu_source Three cleanups from review of AlgorithmConfig: 1. Replace the ECHO-specific `if` in __post_init__ with a data-driven `_ESTIMATOR_DEFAULT_LOSS` map ({ECHO: "echo"}); loss_fn defaults from it when unset. Extensible and not special-cased. 2. Drop the dedicated `env_loss_coef` config field. echo now reads its coefficient from loss_params like every other loss reads its params (e.g. dppo_tv's delta), defaulting to 0.05 in the loss body. Removes the field, its from_config read, the auto-default, the resolve_loss injection, and the base.yaml entry. 3. Remove `mu_source`: it was read in exactly one place — a warning that "proximal" isn't supported on the Fireworks custom path — i.e. a functional no-op. The logp_curr/logp_old/logp_rollout naming now documents the semantics. Tests updated to the new design (estimator→loss default; env_loss_coef via loss_params + loss-body default). 34/34 config + custom-loss tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
`icepop` gates each token on the train/inference ratio d = exp(logp_curr -
logp_rollout): zeros the gradient when d leaves [icepop_alpha, icepop_beta]
(default 0.5/5.0), and weights the score by d inside the band (paper's
M(k)=k·1[a<=k<=b]). Masks out-of-range tokens rather than clipping them.
Uses logp_rollout (the true per-token sampling distribution, correct even across
weight versions in a partial rollout) with logp_curr as the "train" side — no
proximal forward, so it sidesteps the multi-weight-version proximal problem and
works on the default bypass path across all backends. Guards on logp_rollout=None
(loud error, no silent fall back to the proximal logp_old).
Emits icepop/ratio_{mean,min,max} over trainable tokens + mask_frac for
monitoring the discrepancy distribution (batch-true on verl; per-sequence then
averaged on the managed per-datum path). Tightened the logp_rollout docstring;
added to the built-in-losses doc table + a guarded worked example. 29/29 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…egister_loss) Rename register_rllm_adv_estimator -> register_adv_estimator and get_rllm_adv_estimator -> get_adv_estimator, and surface the decorator at the top level as `rllm.register_adv_estimator`, mirroring how `rllm.register_loss` is exposed. This gives custom advantage estimators the same clean, discoverable registration path as custom losses. - rllm/__init__.py: lazy top-level `register_adv_estimator` export - trainer/algorithms/__init__.py: export register_adv_estimator, get_adv_estimator, RLLM_ADV_ESTIMATOR_REGISTRY - trainer/algorithms/advantage.py: rename + keep back-compat aliases (register_rllm_adv_estimator / get_rllm_adv_estimator) so existing custom-estimator imports keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…chat history (#744) Harbor's Chat.chat() drops reasoning_content from assistant history messages unless Terminus2 is constructed with interleaved_thinking=True (Harbor default False), so every request after turn 1 loses all past thinking — bad for reasoning-model distillation, where the captured requests are the SFT context. Plumb the flag through the harness: an interleaved_thinking class attr (env-defaulted from RLLM_TERMINUS_INTERLEAVED_THINKING so `rllm eval` can toggle it without code), a build_env passthrough, and the driver forwarding it to Terminus2(...). Default stays off, matching Harbor. Verified A/B on Fireworks deepseek-v4-pro: with the flag on, every request's history assistant messages carry the exact prior-turn reasoning_content; off, none do. Co-authored-by: Tianhao Wu <44137758+thwu1@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#745) The interleaved_thinking knob reads its env var as the class-attribute default, but enable_summarize was a plain True — and build_env re-exports the attribute into the sandbox, unconditionally overwriting whatever RLLM_TERMINUS_ENABLE_SUMMARIZE the operator set on the host. Eval runs launched with RLLM_TERMINUS_ENABLE_SUMMARIZE=0 therefore still compacted: long trajectories got summarization handoffs ("You are picking up work from a previous AI agent"), breaking the cumulative prefix structure of the recorded chat histories. Read the env var as the default, mirroring interleaved_thinking, and add a build_env round-trip test for both knobs. Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…n fwd/bwd KL to sampler) (#746) Per token, with r = p/q = exp(logp_curr - logp_rollout) against the SAMPLER policy (not a frozen reference): L = -sg[r]*A*logp + bwd_kl_coef*(-log r + r - 1) + fwd_kl_coef*(r*log r - r + 1) Unlike PPO clip (zeros out-of-band gradients) or CISPO/IcePop (clamp/mask the weight), every token always keeps its gradient; the trust region is enforced softly by the KL penalties, using the sample-wise non-negative unbiased estimators under q (k3-style). All knobs are opt-in: fwd_kl_coef / bwd_kl_coef default to 0.0, and ratio_clamp_min / ratio_clamp_max (default None) optionally bound the detached PG weight CISPO-style while the KL terms keep the true r. Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…radient KL forms (#747) Per the latest spec, the KL(p, q) regularizers get a kl_approx switch (default False): - kl_approx=False (new default): score-function forms with the exact KL gradient in expectation under q. Backward: -logp (grad -∇logp = ∇KL(q||p); plain CE on the sampled tokens). Forward: sg[r·log r]·logp (grad r·log r·∇logp = ∇KL(p||q), using E_q[r·∇logp] = 0). - kl_approx=True (previous behavior): Schulman's non-negative unbiased value estimators — backward k3 (-log r + r - 1, grad (r-1)·∇logp), forward (r·log r - r + 1, same per-token gradient as the False form). The reinforce_kl/kl_bwd and kl_fwd metrics always report the estimator values, so logged KLs stay comparable across the knob. Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ailure budget clock (#748) Three ways a recoverable sampling failure was killing whole rollouts: - 429 was not in the transient markers, so rate-limit responses failed the request permanently instead of backing off (routine at high concurrency). - Raw ssl.SSLError (e.g. bad_record_mac) escapes the SDK's SSE stream unwrapped by httpx, so the isinstance network-error check missed it. - The retry budget was measured from REQUEST START, so a long healthy generation that died mid-stream had already spent its own streaming time and got zero retries — systematically failing the longest completions. The clock now starts at the first failure. Also log response headers on final failure (request ids / rate-limit state for provider-side debugging). Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… logs to debug (#749) Two fixes: - Modal small-file uploads went through _exec_unchecked, so a failed write silently no-op'd: the task environment came up broken and its reward-0 trained as a legitimate failure instead of surfacing as SANDBOX_ERROR. Route the write through exec() so callers (setup hooks / evaluator) classify the raise correctly. - Agents fail shell commands constantly as part of normal exploration; logging every nonzero exit at WARNING floods training logs. Demote the per-command failure logs to DEBUG on modal/daytona/local (the exception path still carries the details). Co-authored-by: signalrush <signalrush@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Draft PR to track ongoing terminal-rl work — this branch has accumulated a lot of changes and is 43 commits ahead of
main. Opening as draft to keep the diff visible; will be marked ready / merged once everything settles.What's in here
Cookbooks & recipes
terminal-rlrecipe (Terminus-2 → Terminal-Bench) (feat(cookbooks): add terminal-rl recipe (Terminus-2 → Terminal-Bench) #674)swe-rlrecipe (R2E-Gym → SWE-bench Verified) (feat(cookbooks): add swe-rl recipe (R2E-Gym → SWE-bench Verified) #653)tmaxterminal-agent RL cookbook + tmax-15k datasetCustom losses / algorithms
Fireworks backend
Renderers & engine
TinkerEngine(both backends) (fix(engine): unify renderer rendering+parsing in shared TinkerEngine (both backends) #705)Gateway
rllm tunnelCLI + ngrok backend + daemon auto-pickup (feat(gateway): rllm tunnel CLI + ngrok backend + daemon auto-pickup #695)Sandbox & harness reliability
RLLM_HARNESS_VERIFIER_TIMEOUT_Scap (fix(sandbox): size Modal/Daytona lifetime from agent+verifier timeouts; add RLLM_HARNESS_VERIFIER_TIMEOUT_S cap #736)RLLM_SANDBOX_MAX_{CPUS,MEMORY_MB,STORAGE_MB}resource caps (feat(sandbox): env-var caps for per-task sandbox resources #722)RLLM_HARNESS_RUN_TIMEOUT_Shard ceiling; restore TerminationReason on SandboxedAgentFlow (fix(harness): restore TerminationReason on the SandboxedAgentFlow path #694)Async training
Eval & observability
🤖 Generated with Claude Code