Event cameras stop reading one frame at a time: splitting long streams into multiple frames lifts THU-EACT-50 CHL accuracy from 30.5% to 78.7%

Event-camera action recognition gains clear discriminative power on long, complex actions once the stream is represented as multiple frames, whereas a single frame degrades sharply as sequences grow. Common single-frame representations such as Event Histogram, Time Images and CSTR work well for short event sequences but lose performance on longer sequences with more complex actions. On THU-EACT-50 CHL, CSTR rises from 30.5% with one frame to 47.4% with multiple frames, and Event Histogram from 24.1% to 47.1%.

The event stream is divided into non-overlapping temporal segments, each segment is turned into its own frame-based representation and tokenized separately, and tokens are then ranked by event activity so the most informative ones are kept. Multiple frames amplify the sparsity already present in event data and increase the number of input tokens, so naively adding frames would raise computational cost. On THU-EACT-50 CHL, halving the token budget with a single frame improves accuracy from 29.4% to 46.5%, and keeping only half the tokens is also best with two frames.

Pre-training uses sparsity-aware masking, which raises the probability that high-activity patches stay visible as context, and the decoder is initialized with multiple position-dependent mask tokens. Standard random masking ignores token content, so at a 75% masking ratio the decoder's context can be nearly empty, and that risk grows with the number of frames. In linear probing on THU-EACT-50 CHL, sparsity-aware masking adds roughly 4 percentage points for both CSTR and Event Histogram, and multiple mask tokens beat a single one for both representations.

The method targets long, complex event-camera action recognition and is evaluated on THU-EACT-50 CHL and DailyDVS-200 with linear probing and fine-tuning, with pre-training and evaluation on the same dataset by default and one transfer setting that pre-trains on DailyDVS-200 and fine-tunes on THU-EACT-50 CHL. A reader interested in short sequences or other event representations can first check whether multi-frame splitting and activity-based token selection still help.

No single optimal number of mask tokens was found across all experiments, and performance varies with the sparsity bias parameter and the event representation used, so the authors see a need for more investigation before general recommendations. Going from four to eight frames slightly improves the short action sneeze while slightly reducing accuracy for the long action tie shoe laces, indicating that the effect of temporal resolution can differ between actions.