[Question] Qwen3.5 Support (e.g., Qwen3.5-35B-A3B)

[Yanbin-Yin]

Qwen3.5 Support (e.g., Qwen3.5-35B-A3B)

Question

I'm trying to use slime for RL training on Qwen3.5-35B-A3B and would like to check the current support status.

I noticed that slime already has scripts for Qwen3-30B-A3B (scripts/run_qwen3_30b_a3b.py), and since Qwen3.5-35B-A3B shares a similar MoE architecture (also with ~3B active parameters per token), I'm wondering whether it's already supported or if there are known compatibility issues.

Background

Qwen3.5 introduces some architectural changes compared to Qwen3, most notably:

• Gated Delta Networks (a linear attention variant) interleaved with standard attention
• Updated MoE routing
• Native multimodal (early-fusion vision-language)

These differences may affect compatibility at the SGLang rollout layer and/or the Megatron-LM training layer.

Specific Questions

1. Is Qwen3.5 (e.g., Qwen3.5-35B-A3B) currently supported, or is it on the roadmap?
2. Is it safe to reuse scripts/run_qwen3_30b_a3b.py with a Qwen3.5 checkpoint by simply swapping the --hf-checkpoint path, or are additional config changes required?
3. Does the current SGLang version bundled with slime support Qwen3.5's hybrid attention (Gated Delta Networks)?
4. Any known issues or workarounds for running Qwen3.5 MoE models?

Environment

• slime version: latest main
• Model: Qwen/Qwen3.5-35B-A3B
• GPU: NVIDIA (CUDA)

Thanks for the great work on slime! Happy to test and provide feedback if Qwen3.5 support is in progress.

What I've Tried

Null

Environment (if relevant)

• slime version:
• Python version:
• PyTorch version:
• CUDA/ROCm version:
• GPU type and count:
• OS:

Additional Context

No response

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.

[cjy0x]

you can check examples/geo3k_vlm folder @Yanbin-Yin

[VincentWangzt]

@cjy0x the example code at geo3k_vlm seems broken, running the script will result in a config error:

Traceback (most recent call last):
File "/root/slime/train_multi_model.py", line 272, in
args = parse_args()
^^^^^^^^^^^^
File "/root/slime/slime/utils/arguments.py", line 1431, in parse_args
slime_validate_args(args)
File "/root/slime/slime/utils/arguments.py", line 1574, in slime_validate_args
if not os.path.exists(args.ref_load):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 19, in exists
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

[none0663]

Qwen3.5 Support (e.g., Qwen3.5-35B-A3B)

May you can try this script below

#!/bin/bash
set -e
set -p
set -x
unset http_proxy
unset https_proxy
# will prevent ray from buffering stdout/stderr
export PYTHONBUFFERED=16

NVLINK_COUNT=$(nvidia-smi | grep -o "NVLink" | wc -l)
if [ "$NVLINK_COUNT" -gt 0 ]; then
    HAS_NVLINK=1
else
    HAS_NVLINK=0
fi
echo "HAS_NVLINK: $HAS_NVLINK (detected $NVLINK_COUNT NVLink references)"

SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"

echo "SCRIPT_DIR: $SCRIPT_DIR"

# unset proxy to avoid issues
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY

source "/root/slime/scripts/models/qwen3.5-35B-A3B.sh"

hf_model_path="Qwen/Qwen3.5-35B-A3B"
CKPT_ARGS=(
   --hf-checkpoint $hf_model_path
   --ref-load Qwen3.5-35B-A3B-dist 
   --load checkpoint_slime/Qwen_35_2026
   --save checkpoint_slime/Qwen_35_2026
   --save-interval 100
)

ROLLOUT_ARGS=(
   --prompt-data ${SCRIPT_DIR}/dapo-math-17k/dapo-math-17k.jsonl
   --input-key prompt
   --label-key reward_model
   --apply-chat-template
   --rollout-shuffle
   --num-rollout 5000
   --rollout-batch-size 128
   #--over-sampling-batch-size 192
   --n-samples-per-prompt 8
   --rollout-max-response-len 5200
   --rollout-max-prompt-len 10000
   --rollout-temperature 1

   --global-batch-size 256
   --balance-data
)

EVAL_ARGS=(
   --eval-interval 20
   --eval-prompt-data \
     aime aime2025.parquet
   --eval-input-key prompt
   --eval-label-key reward_model
   --eval-max-response-len 12384
   --eval-max-prompt-len 1024
   --eval-top-p 0.3
)

PERF_ARGS=(
   --tensor-model-parallel-size 2
   --sequence-parallel
   --pipeline-model-parallel-size 2
   --context-parallel-size 1
   --expert-model-parallel-size 8
   --expert-tensor-parallel-size 1

   --recompute-granularity full
   --recompute-method uniform
   --recompute-num-layers 1

   # --micro-batch-size 1
   --use-dynamic-batch-size
   --max-tokens-per-gpu 16386
   #--max-tokens-per-gpu 5120
)

GRPO_ARGS=(
   --advantage-estimator grpo
   --use-kl-loss
   --kl-loss-coef 0.00
   --kl-loss-type low_var_kl
   --entropy-coef 0.00
   --eps-clip 0.2
   --eps-clip-high 0.28
   --use-tis
)

OPTIMIZER_ARGS=(
   --optimizer adam
   --lr 5e-7
   --lr-decay-style constant
   --weight-decay 0.1
   --adam-beta1 0.9
   --adam-beta2 0.98

   --optimizer-cpu-offload
   --overlap-cpu-optimizer-d2h-h2d
   --use-precision-aware-optimizer
)

WANDB_ARGS=(
   #--use-wandb
   #--wandb-project slime-dev-32-gpu
   #--wandb-group qwen3-32B-32gpus
   # --wandb-key ${WANDB_KEY}
   #--wandb-mode offline
)

SGLANG_ARGS=(
   --rollout-num-gpus-per-engine 8
   --sglang-mem-fraction-static 0.70
   --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 256)
   --sglang-ep-size 8
   # --sglang-enable-ep-moe
   # mtp
   --sglang-speculative-algorithm EAGLE
   --sglang-speculative-num-steps 2
   --sglang-speculative-eagle-topk 1
   --sglang-speculative-num-draft-tokens 3
   #--sglang-enable-draft-weights-cpu-backup

   --sglang-max-running-requests 512
   #--use-slime-router
   #--use-rollout-routing-replay
   #--use-routing-replay
   #--partial-rollout
)

MISC_ARGS=(
   # default dropout in megatron is 0.1
   --attention-dropout 0.0
   --hidden-dropout 0.0
   # should be good for model performance
   --accumulate-allreduce-grads-in-fp32
   --attention-softmax-in-fp32
   # need to comment this when using model with MLA
   --attention-backend flash

   --moe-token-dispatcher-type flex
   #--moe-enable-deepep
)

# Build the runtime environment JSON with proper variable substitution
RUNTIME_ENV_JSON="{
  \"env_vars\": {
    \"PYTHONPATH\": \"/root/Megatron-LM/",
    \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\",
    \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\"
  }
}"


current_date=$(date +"%Y_%m_%d")

cur_path=`pwd`

ray job submit --address="http://${MASTER_IP}:8265" \
    --runtime-env="${cur_path}/runtime_env.yaml" \
    -- \
    NCCL_DEBUG=INFO,PYTHONUNBUFFERED=1 python3 /root/slime/train.py \
   --actor-num-nodes 4 \
   --actor-num-gpus-per-node 8
   --colocate \
   --offload \
   --no-check-for-nan-in-loss-and-grad \
   --distributed-timeout-minutes 190 \
   --use-tensorboard \
   --tb-project-name 35-35B-80B \
   --tb-experiment-name n_12 \
   ${MODEL_ARGS[@]} \
   ${CKPT_ARGS[@]} \
   ${ROLLOUT_ARGS[@]} \
   ${OPTIMIZER_ARGS[@]} \
   ${GRPO_ARGS[@]} \
   ${DISTRIBUTED_ARGS[@]} \
   ${WANDB_ARGS[@]} \
   ${PERF_ARGS[@]} \
   ${EVAL_ARGS[@]} \
   ${SGLANG_ARGS[@]} \
   ${MISC_ARGS[@]} 2>&1 | tee xxxx_${current_date}.log &

[VincentWangzt]

Yeah it works, thanks! I figured out my own script as well lol, just documenting it here.
follow the script in scripts/run-qwen3.5-xxx, the ones at examples/geo3k_vlm is broken, missing ref-load as well as using bshd attention format instead of dynamic batch size.
Anyway, it's just the errors and some indiscrepancies I encountered along the way.