Skip to main content
Back to timeline
arXivSource publication:

LACS moves the pruning cutoff with the current confidence spread: 29% fewer backward rows on CIFAR-10 at 89.77% versus 89.82% accuracy

Synopsis

Which examples receive backward computation during training no longer has to follow a preset quota or an absolute confidence cutoff. LACS compares each candidate's exponentiated maximum predictive confidence with the current population mean of exponentials, clips the coefficient at zero, and thereby applies a cutoff equal to a log-mean-exp reference plus a margin, with the retained count following the score distribution. On CIFAR-10 it removes 29% of backward rows relative to full cross-entropy at accuracy means of 89.77% versus 89.82%, with a three-seed interval crossing zero; on 14 tabular datasets quadratic boosting lowers mean test error from 23.36% to 22.96%.

Source-provided article image: LACS: Distribution-Sensitive Label-Agnostic Confidence Selection for Dynamic Data Pruning
arXiv

Interpretation

Which examples enter backward propagation is now decided by the current batch's confidence distribution, which sets both the threshold and the retained count. A fixed quota fixes the amount removed before the score gaps are examined, and an absolute confidence cutoff ties the decision to the score scale; neither moves with the current distribution. On CIFAR-10 with ResNet-32, LACS removes 29% of backward rows relative to full cross-entropy, with accuracy means of 89.77% versus 89.82% and a confidence interval that crosses zero over three seeds.

The rule reads only maximum predictive confidence, so selection needs neither labels nor per-example gradients. Each candidate's exponentiated maximum confidence is compared with the current population mean of exponentials, and clipping the coefficient at zero yields a cutoff equal to a log-mean-exp reference plus a margin. Across 14 tabular datasets, quadratic weak learners lower mean test error from 23.36% to 22.96%, and full training replays use about 30% fewer fitting rows.

Saved computation is not the same as saved time, and the two are counted separately. The current neural implementation runs a no-gradient scoring forward to obtain fresh scores, then a forward and backward on retained examples only, so backward sparsity does not become wall-clock acceleration. On one host LACS is slower than full cross-entropy, while a 300-round RBF-SVM replay on credit data fits 29.3% fewer rows, cuts end-to-end time by 37.2%, and slightly lowers final error.

Perspective

The demonstrated scope is seven evaluation settings: supervised neural training, Transformer language modeling, long-tailed classification, semi-supervised learning, and boosting with compact nonlinear, tree and kernel-SVM weak learners; neural head-to-head comparisons are limited to CIFAR-10 and CIFAR-100 with three seeds each, tree comparisons use the fixed-threshold companion rule LACS-T, and the RBF-SVM timing evidence rests on four dataset/split cells with nearly equal round counts.

On CIFAR-10 the gap to InfoBatch-B is positive in all three seeds but statistically unresolved, and the comparison with InfoBatch is also unresolved; the current two-pass neural implementation is slower than both full cross-entropy and InfoBatch; on Transformer language modeling, selective execution reaches a validation NLL of 1.5688 versus 1.5666 while remaining 4.5% slower than the optimized dense implementation; long-tailed and semi-supervised settings provide boundaries rather than positive evidence.