Skip to content

fix(mlx): tolerate invalid UTF-8 VLM tokens - #5382

Open
accelizero wants to merge 1 commit into
opendatalab:masterfrom
accelizero:fix/mlx-vlm-utf8-compat-cla
Open

accelizero wants to merge 1 commit into
opendatalab:masterfrom
accelizero:fix/mlx-vlm-utf8-compat-cla

Conversation

@accelizero

Copy link
Copy Markdown

Motivation

Older mlx-vlm releases strictly decode the pending byte-level BPE buffer. A generated byte-fallback token followed by a space-prefixed token can raise UnicodeDecodeError and abort the full MinerU VLM task.

Modification

  • Backport the upstream replacement-decoding behavior for older mlx-vlm releases at MLX engine initialization.
  • Leave releases that already use errors="replace" unchanged.
  • Add focused regression coverage for both paths.

Validation

  • Focused Ruff check passes for the new compatibility module and regression test.
  • Regression tests pass: 2 passed.
  • Verified a one-page Hybrid MLX parse on macOS with MinerU 3.4.4, mlx-vlm 0.3.9, mlx 0.31.1, and transformers 4.57.6.

BC-breaking

No. The patch only prevents an MLX VLM generation crash; non-MLX backends and already-fixed mlx-vlm versions are unchanged.

This branch is based on the locally tested Git commit. The earlier web-editor PRs were closed because their new files had incorrect nested paths.

@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 10, 2026
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@dosubot dosubot Bot added the bug Something isn't working label Aug 10, 2026
@accelizero

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@accelizero

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant