Action Recognition: new task, model ABC, and workflow block

[leeclemnet]

What does this PR do?

Adds Action Recognition: a video task that assigns class labels to frame ranges, and the workflow block that runs it over a stream. Output is (start_frame_idx, end_frame_idx, class) tuples, which can overlap. Net diff: 30 files, +4506/−23.

One output contract covers both kinds of checkpoint, so the block never knows which it holds. A zero-shot model returns a verdict for the whole window. A fine-tuned model returns spans inside the window.

Model layer (inference_models)

• New task action-recognition, the ActionRecognitionModel ABC, and the frozen ActionRecognitionPrediction result. A VideoSampling contract travels with the model and carries window length, sample rate, minimum frames, sampling mode, frame budget, and trained frame side.
• Cosmos3EdgeActionRecognition implements the ABC over the pure-VLM reasoner and picks its mode from the package. A tokenizer with <|cls:...|> tokens is a fine-tune; anything else runs zero-shot.
• Zero-shot: one call per window, greedy, thinking off, 256 tokens, native resolution. With classes it asks a lettered multiple choice and reads the letter. Without classes it asks for one short phrase. About 0.7 s per call on an L4 at 2 s windows.
• Fine-tune (packages from roboflow-train#880/Outlier Detection Block #884): the training contract verbatim — SFT system prompt, class-token legend, and <|cls:name|> <S.SS> <E.EE> span lines. Decoding runs through a prefix_allowed_tokens_fn state machine that admits only tokens continuing a valid line, so the answer always parses. Frames are capped at the trained side. Classes come from class_names.txt, and sampling from inference_config.json. A package with class tokens but no class list raises CorruptedModelPackageError rather than loading as zero-shot.
• The reasoner gained LoRA package loading: base weights under base/, the class-token tokenizer on top, and an embedding resize before the adapter attaches. It also gained enable_thinking and constrained-decoding support on the video path, and real VideoMetadata so the model's frame timestamps match the clip.

Workflows layer (inference)

• New block roboflow_core/roboflow_action_recognition_model@v1 and a tensor sibling. model_id is the standard model field; class_filter restricts a fine-tune's classes or forms the zero-shot vocabulary.
• Window length and sample rate are training decisions, so the block reads them from the model. stride_seconds is the only temporal setting, and it defaults to the window, which tiles the stream without overlap.
• The timeline only grows: same-class ranges merge when no sampled frame lies between them, and the block never extends a range on its own. window_classes lists classes that a later call can still merge with, and error_status reports a failed call without ending the stream.
• Blocks now classify what they still hold when a stream ends, through the existing stream-pipeline flush. Without it a clip shorter than one window produced nothing. The WebRTC worker flushes at the point where it reports completion, and it now reads the handler result correctly when a workflow holds a pipelined block.
• New kind action_recognition_prediction with its serializer and deserializer, and ffmpeg in the dev GPU image, without which upload FPS detection failed silently.

Known limit. Zero-shot classifies a whole window, so its boundaries land on window edges. In-window localization is the fine-tune path.

Related Issue(s): consumes roboflow-train#880 and #884.

Type of Change

• New feature (non-breaking change that adds functionality)

Testing

• I have tested this change locally
• I have added/updated tests for this change

Test details:

Unit tests, all green in the CUDA staging container: 51 in the model layer (mode routing, lettered and phrase parsing, the fine-tune prompt and span format, per-call class subsets, the constrained-decoding automaton, the package readers, the malformed-package error, per-mode frame resolution) and 76 on the block (manifest, fire cadence, float sampling grid, FPS pinning, monotone merging, keep-alive window_classes, whole-video mode, end-of-stream flush, resets, tensor paths). The stream handler suite adds 23, and the visualization suite 14.

Live validation on an L4, through the workflow path with real weights:

• A fine-tuned checkpoint on its 15.5 s training clip (labels: out 0–8 s, in 8.5–15.5 s) returns out and in with boundaries inside half a second of the labels, at the 8 s window its package declares. A 6 s clip, shorter than that window, is classified whole by the end-of-stream flush; before the flush it returned nothing.
• Zero-shot on a 21.5 s traffic clip detects the overtake with no false positives across about 40 calls, and window_classes stays on for the event and clears when the range closes.
• Both tensor and numpy paths run, with ENABLE_TENSOR_DATA_REPRESENTATION on and off.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

The prompt and decode settings come from measurements against NVIDIA's own demo assets, not preference. Zero-shot cannot localize inside a window under a cheap decode: the SFT span contract, the cookbook dense-caption prompt, and the guide's timestamp query all collapse to positional templates without a think block, and neither a larger token budget, nor thinking, nor NVIDIA's sampling parameters, nor lower resolution recovers it. Thinking-on dense captioning does localize, at 20–90 s per call, which is why an earlier two-stage design was dropped. The fine-tune trains that ability into the cheap path, so the shipped zero-shot mode buys classification and the fine-tune buys boundaries.

Testing ran on an L4 staging build. Implementation used Codex agents for two of the larger changes, with review and live validation here.

🤖 Generated with Claude Code

[github-actions]

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚧 Right now this is a draft, so automated Claude review is paused — nothing is being spent yet. Mark it Ready for review to trigger it.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

• 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
• 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

• 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
• ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
• 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
• 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

• Prefer to skip automated review entirely? Add the skip-claude-review label.