A nice UX for generating dynamic tensors that break torch.compile/torch.export

[IlyasMoutawwakil]

Feature request

We have a couple tensors whose construction is highly dynamic and can't be captured while tracing (cu_seqlens, vision(image/video)_cu_seqlens, ... , vision(image/video)_position_ids, ...). I'm opening this issue to track what we are gonna do about them. The two options are either using the processor since that's where we have the most of dynamism / per-sample processing loops. or unifying the collator API (processor - > collator -> inference) as part of the default path.

Motivation

see #45396

Your contribution

can implement it :)

[zeel2104]

Is it open for contribution?

[IlyasMoutawwakil]

open for ideation first 😆 see #45396 for more details

[zeel2104]

Hey @IlyasMoutawwakil
Thanks for linking #45396. I read through the PR and it looks like the core model-side work is already being handled there: dynamic vision/audio tensors are extracted into pure helpers and accepted as optional precomputed inputs.

For #45571, would a useful follow-up be to focus specifically on the user-facing UX after #45396 lands?

My current understanding is that the remaining design question is where users should get these tensors from:

1. processor output directly, e.g. processor(..., return_extra_tensors=True)
2. a data collator that turns processor outputs into compile/export-ready model inputs
3. hidden/internal kwargs so normal users do not see these power-user tensors

I’d be happy to help with a small follow-up PR once the direction is decided. A possible first scoped task could be:

• pick one model family, e.g. Qwen2-VL/Qwen3-VL
• add or refine processor/collator support for returning the precomputed tensors
• add a compile/export-oriented test or example
• keep backward compatibility when tensors are not passed

Would maintainers prefer the processor-only path, or should I explore the processor -> collator -> model path?