Gated spiking cross-attention puts the gate before spike firing, lifting CREMA-D accuracy by 2.15 points
Synopsis
Spiking networks that fuse audio and video have lacked a way to regulate information flow inside cross-modal attention according to the input. GSCANet adds a gate to a dual-stream Spikingformer backbone, placing it on the attention output just before spike firing and driving it with concatenated query and value features. On CREMA-D and RAVDESS this configuration beats the no-gate baseline by 2.15 and 1.05 percentage points; both gate position and driving signal change the outcome.
Interpretation
An audiovisual spiking network now has an input-dependent gate that sets how strongly cross-modal information flows, placed after attention aggregation and before spike firing. Earlier spiking cross-attention projected and aggregated query, key and value features and passed the result straight to a spiking neuron, with no dedicated mechanism deciding how strongly the current cross-modal representation should shape the next spike response. On the same backbone, attention-output gating raises accuracy from 76.34% without a gate to 78.49% on CREMA-D, and from 80.70% to 81.75% on RAVDESS.
Each attention head has its own small multilayer perceptron that produces the gate, whose output is squeezed between 0 and 1 by a Sigmoid and multiplied element-wise into that head's accumulated attention result. The driving signal concatenates query and value: the query carries what the primary modality needs, the value carries what the auxiliary modality supplies, and both enter the gate generator. Across all four datasets the joint query-value driver beats query-only and value-only; on CREMA-D it is 3.36 percentage points above query-only.
Perspective
The demonstrated scope is audiovisual classification: two synchronized emotion datasets, CREMA-D and RAVDESS, plus the category-paired UrbanSound8K-AV and CIFAR10-AV, all trained in the Brain-Cog framework on a single A100 with 4 simulation time steps.
Gating does not improve every placement or input condition: block-output gating collapses to a constant 0.5 on RAVDESS and scores below the no-gate baseline. In the modality-failure test, masking audio raises video-only accuracy by 2.69 points on CREMA-D, but masking vision lowers audio-only accuracy by 1.08 points on CREMA-D, and both directions drop on RAVDESS. The authors treat the gate as representation-level modulation rather than a missing-modality recovery mechanism.