vllm_plugin: `_merge_mm_kwargs` IndexError at predict time with vllm 0.21.0 (cannot serve VibeVoice-ASR via vllm)

[Alfarouk-Baseten]

Summary

Cannot deploy microsoft/VibeVoice-ASR via vLLM (any 0.10.x – 0.21.0 tested) using the bundled vllm_plugin. The plugin has multiple API drifts vs. published vLLM releases, and the deepest blocker is an IndexError: list index out of range inside vLLM's _merge_mm_kwargs during inference — this fires even after working around earlier issues with --skip-mm-profiling, a custom chat-template content format, and patching the plugin's get_data_parser method.

Filed in case there's a known-good combo / patch on your side, or as a heads-up for plugin maintenance work.

Environment

• Base image: vllm/vllm-openai:v0.21.0 (also tested v0.14.1 which docs/vibevoice-vllm-asr.md mentions)
• Plugin: git+https://github.com/microsoft/VibeVoice.git@main (commit pulled fresh)
• Model: microsoft/VibeVoice-ASR
• GPU: NVIDIA H100 80GB
• Deployment platform: Baseten Truss (but the underlying vllm serve command is canonical)

Repro

Run vllm serve microsoft/VibeVoice-ASR with Microsoft's documented flag set from vllm_plugin/scripts/start_server.py (--no-enable-prefix-caching, --enable-chunked-prefill, --chat-template-content-format openai, --allowed-local-media-path /app, --trust-remote-code, --dtype bfloat16).

POST /v1/chat/completions with the multimodal payload from vllm_plugin/tests/test_api.py:

{
  "model": "vibevoice",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant that transcribes audio input into text output in JSON format."},
    {"role": "user", "content": [
      {"type": "audio_url", "audio_url": {"url": "data:audio/mpeg;base64,<...>"}},
      {"type": "text", "text": "This is a 10.26 seconds audio, please transcribe it..."}
    ]}
  ],
  "max_tokens": 32768,
  "temperature": 0.0
}

What we hit and how we (partially) worked around

#	Issue	Workaround
1	Startup: vLLM's automatic multimodal memory profiler crashes with IndexError in _initialize_kv_caches → determine_available_memory → profile_run → _merge_mm_kwargs. Triggered by plugin's audio encoder returning malformed output on synthetic dummy audio.	--skip-mm-profiling
2	Predict: chat template applies `'<	im_start
3	Predict: RuntimeError: Audio resampling is not supported when target_sr is not provided from vllm/multimodal/audio.py:AudioResampler.resample. Root cause: plugin defines _get_data_parser (underscored) on VibeVoiceMultiModalProcessor, but vLLM v0.21 calls info.get_data_parser() (no underscore) on the Info class. The plugin's method is dead code.	Added get_data_parser to VibeVoiceProcessingInfo returning MultiModalDataParser(target_sr=24000)
4	Predict: IndexError: list index out of range in vLLM's _merge_mm_kwargs at missing_updates_item = missing_prompt_updates[missing_next_idx]. Plugin's _get_prompt_updates returns a single PromptReplacement(modality=\"audio\", ...) — apparently the wrong shape for vLLM v0.21's prompt-update merging contract.	Currently blocked.

Item 4 is the remaining blocker. The single-PromptReplacement return shape worked for some earlier vLLM version but not v0.21.

Other API drift observations

vllm_plugin/model.py also uses APIs from different vLLM eras simultaneously:

• ProcessorInputs(mm_data=...) — only in v0.10-v0.14.1; renamed to mm_data_items (with type MultiModalDataItems) in v0.15+
• _get_dummy_audios(overrides=...) — only in v0.12+

The intersection where both work is v0.12-v0.14.1, but those minor versions have no published vllm/vllm-openai Docker image (the registry jumps from v0.10.1.1 directly to v0.15.0+). v0.14.1 is documented in docs/vibevoice-vllm-asr.md but produces the same Layer-1 startup IndexError without --skip-mm-profiling.

Asks

1. Is there a known-good combination of vLLM version + plugin commit you ship internally that we're missing? (No Dockerfile is published in the repo, and pyproject.toml doesn't pin vLLM.)
2. Is the v0.21 prompt-update shape mismatch (item 4 above) something the maintainers are tracking? Happy to help test a fix.
3. Would it be valuable for the repo to:
   • Publish a Dockerfile + tested vLLM version pin (e.g. in pyproject.toml [tool.vibevoice.compat] or a requirements-vllm.txt)
   • Add a docs/vibevoice-vllm-asr.md section noting which vllm/vllm-openai tag has been validated end-to-end

[pengzhiliang]

Thanks for the detailed report.

For vLLM 0.21.0, the current plugin has not been validated yet. vLLM has changed several multimodal APIs across versions, so 0.21.0 compatibility will need a separate update.

For the documented path, we currently validate against vllm/vllm-openai:v0.14.1 together with our launcher script:

python3 /app/vllm_plugin/scripts/start_server.py

Please make sure the 0.14.1 test uses the full launcher rather than manually calling vllm serve with copied flags, because the launcher also installs the package, downloads the model snapshot, and generates the tokenizer files before starting vLLM.

If the exact documented docker command with vllm/vllm-openai:v0.14.1 still fails, could you share the full startup log, including the vLLM version, tokenizer generation step, and the first traceback? We will investigate that as a regression in the supported deployment path.

Also, PRs to adapt the plugin to newer vLLM releases are very welcome. We are tagging this as a compatibility issue so we can track newer-vLLM support separately from the currently validated deployment path.