Skeleton action recognition gets a quantization plug-in: white-box attack success drops from near 100% to single digits, accuracy stays comparable
Synopsis
Mainstream contrastive models for self-supervised skeleton-based action recognition had almost no defense against adversarial attacks, where tiny perturbations to skeleton data could sharply degrade recognition. QCLR adds quantization as a plug-in module at two levels of the encoding pipeline, mapping continuous inputs and features to discrete codebooks so small perturbations are absorbed and gradient propagation is broken; integrated into SkeletonCLR, AimCLR and ActCLR on NTU-60, NTU-120 and PKU-MMD, it cuts PGD and SMART attack success rates to single digits while keeping classification accuracy comparable to the original models.
Interpretation
Skeleton action recognition models are exposed to adversarial perturbations when skeleton data is transferred, and QCLR turns defense into a plug-in that leaves the original training objective unchanged, making models steadier against both white-box and black-box attacks. No defense had been designed specifically for self-supervised skeleton-based action recognition; only general defenses such as TRADES or the supervised defense BEAT were available, and applying them to mainstream contrastive models yielded limited robustness gains. On NTU-60, NTU-120 and PKU-MMD, integrating QCLR into SkeletonCLR, AimCLR and ActCLR lowered PGD and SMART attack success rates from near 100% to single digits, while APGD-DLR also stayed lowest.
Two-level quantization discretizes both input and features across the encoding pipeline, making it hard for attacks to accumulate perturbations along gradient directions. Input quantization alone fails against PGD and SMART, and feature quantization alone fails against APGD-DLR and BASAR; only their combination covers multiple attack types. Ablation shows input quantization cut APGD-DLR success from 99.42% to 59.93%, feature quantization cut PGD success from 99.98% to 5.94%, and combining them suppressed both.
Attention-guided multi-faceted quantization uses multiple codebooks to share semantic facets, easing the representation degradation that vector quantization inevitably brings. Naive vector quantization under a 4096-codeword limit either diverged contrastive learning to near-random accuracy or still lost significant accuracy when training stayed continuous. On NTU-60, naive implementations dropped SkeletonCLR accuracy to 1.67% or 42.55%, while multi-faceted quantization held 72.95%, above the original model's 68.3%.
Perspective
Evaluation covers NTU-60, NTU-120 and PKU-MMD under the cross-subject setting, where training and test data come from different subjects. Defense is validated on three contrastive models, SkeletonCLR, AimCLR and ActCLR, with inference at about 433 samples per second, and a single TITAN RTX provides sufficient memory.
Vector quantization inevitably introduces quantization error, and multi-faceted quantization only mitigates rather than removes it; the input quantization step size sets the accuracy-robustness trade-off, with smaller steps giving higher accuracy but weaker defense. The black-box attack BASAR always reaches 100% success by design, so its effect is measured by joint position, acceleration and bone-length deviations of the generated samples.