Video action explanations split into what the model sees and when it moves: lowest deletion AUC on most axes across three datasets, with masks about five times sparser
Synopsis
Explanations for video action recognition usually fold spatial appearance and temporal motion into a single saliency map, so a user cannot tell whether the model predicted from the object or from the movement. This work separates the explanation into a spatial appearance mask and a temporal motion mask, optimized jointly against a frozen classifier, with nearest-frame replacement used to remove temporal dynamics. On EPIC-Kitchens-55, EGTEA Gaze+, and Something-Something V2, the method attains the lowest deletion AUC on most datasets and axes, with masks roughly one-fifth the size of the strongest baseline.
Interpretation
An explanation for video action recognition can now be split into two masks: one marking the pixels that sustain the noun prediction, one marking the frames that sustain the verb prediction. Earlier gradient-based and perturbation-based video explanation methods return a single spatio-temporal mask, mixing spatial pixel and temporal frame importance in one map, so a reader cannot tell which kind of evidence the model relies on. On EPIC-Kitchens-55, appearance deletion AUC falls to 0.10 against 0.15 for Extremal Perturbation and 0.23 for STEP, and motion deletion AUC falls to 0.21 against 0.50 for the temporal extension of Extremal Perturbation.
Both masks act on the frozen classifier through one compositional perturbation forward pass: the appearance mask controls whether pixels are blurred or kept, and the motion mask controls whether frames are replaced or kept. On the temporal axis the perturbation uses nearest-active-frame replacement, so frozen frames copy the content of a nearby kept frame and temporal dynamics go to zero rather than being attenuated; the replacement source is the spatially perturbed clip, which propagates the appearance mask to the replacement targets. Evaluation covers EGTEA Gaze+, EPIC-Kitchens-55, and Something-Something V2 with frozen TSM and TimeSformer backbones, compared against STEP, Extremal Perturbation, and a single-mask ablation.
The appearance mask is confined to the frames the temporal mask keeps, so budget is not spent on positions the model never sees. Without this constraint the optimizer is free to place appearance budget on frames that have been replaced and therefore never reach the model input. With the separation penalty and the ReLU upper-bound area constraint, the appearance mask retains only 10.3% of pixels and the motion mask only 24.5% of frames on EPIC-Kitchens-55, whereas STEP retains 66.1% of pixels and uses every frame.
Perspective
Evaluation covers the EPIC-Kitchens-55, EGTEA Gaze+, and Something-Something V2 datasets with frozen TSM and TimeSformer backbones; the explanation targets an already-trained video action classifier, and the method does not change classification accuracy.
Factoring strictly by appearance and motion is inherently restrictive and unsuitable for actions defined by audio cues, gaze patterns, off-frame interactions, or long-range interactions lasting more than 8 frames; single-head models rely on optical-flow alignment as a pragmatic prior, and measured cross-axis gradients are of the same order as the intended ones, so semantic separation is an empirical outcome rather than a structural guarantee.