TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions

Ground truth and three detectors, all at 25 fps, on the same benchmark videos. Every clip here is a transition TransVLM detects and both shot-boundary detectors miss, cut out of the video it came from. 84 clips in total. Drag the timeline to move through one.

FFmpeg’s 58 native xfade transitions and the cut: the whole vocabulary of the data engine behind the synthetic half of the benchmark and most of the training set. Point at a tile, or tap it, and it plays with its definition.

For each pair of adjacent shots the engine draws one effect uniformly from the 59 and a duration uniformly up to a bound set by the two shots. A cut forces the duration to zero. The shots themselves are mined by running PySceneDetect, TransNetV2 and AutoShot and keeping only those a majority agree on, so the pool carries almost no residual transitions.

Long transitions are the easiest to detect because several seconds of surrounding context anchor both endpoints. Normal-length transitions in the public set are the most difficult: the model tends to overpredict them, increasing false positives and lowering F1. Cuts are usually detected but localized one frame off. On synthetic cuts, the average boundary error is 0.04 second, which yields an almost negligible IoU because cuts have no temporal duration. In the public subset, some long transitions are detected, but their endpoints can be off by as much as two seconds, where the ground-truth boundaries are themselves least certain.

The remaining errors occur mainly in very low-light scenes, where both colour and optical flow become unreliable, and in long transitions that span two sliding windows and lose temporal coverage at the boundary between them.

VLMs capture spatial semantics effectively but struggle with motion across shot boundaries. Instead of adding a separate encoder, TransVLM expands the vision patch embedding from three to six channels, initializes the new weights to zero, and processes concatenated colour and optical-flow inputs.

The visual token sequence does not get longer, so the language backbone carries no additional cost.

Public SBD datasets represent cuts as timestamps, making them unable to capture transitions such as a two-second dissolve. The STD benchmark addresses this by using six datasets with segment-level annotations (three re-annotated frame by frame for this study and three adopted as provided) alongside synthetically generated videos.

Transitions are grouped by duration — Cut under 0.1 s, Normal up to 1 s, Long beyond that.

Expert annotators worked through a frame-precise tool, one primary pass plus at least one reviewer, disagreements arbitrated to a single consensus label. We will release the benchmark annotations after our internal inspection process.

Across four of the six datasets, the public annotations omit 14 transitions entirely. Most are gradual transitions, the very kind of event that point-based labels cannot represent.

No baseline is strong on both halves of the benchmark. Public data is about 80% cuts, which the shot-boundary detectors were built for and which a VLM fed five frames a second cannot resolve; the synthetic half is mostly gradual, and there the order reverses. TransVLM is the only row that holds on both.

Same frame rate. The zero-shot VLMs were run at 5 fps. Run at 5 fps itself, TransVLM still leads them on frame-level F1 on both halves and on segment-level F1 on the synthetic half; on public segment-level F1 Gemini 3 Pro is ahead by 0.001.

Well-formed output. Untuned, Qwen3-VL-4B returns unparseable JSON for 4.96% of its outputs; TransVLM for 0.03%. Take the sliding window away, so the whole video goes through one forward pass, and the rate climbs back to 4.35% while public segment-level F1 falls to 0.540. The 2B variants fail on 54% and 65% of outputs, which is why 4B is the smallest backbone used.

Where flow helps. The ablation view shows what removing optical flow costs overall. On the synthetic subtle-cut split alone, cuts with little change in content, flow lifts segment-level F1 from 0.681 to 0.732 and frame-level F1 from 0.555 to 0.642: it is the abrupt, low-contrast cuts it recovers.