Blockwise distillation search breaks at a 55% latency budget: pairwise search leads by 3.3–16.6 points
Synopsis
Blockwise distillation architecture search picks child networks by summing each replacement block's isolated score, an additive assumption rarely tested directly. On DeiT-S and DeiT-B, as the block-latency budget drops from 60% to 55%, the additive integer linear program begins to trail a pairwise integer quadratic program that models block interactions, by 3.3–16.6 top-1 points on ImageNet-100 after fine-tuning. The error splits into a metric part from the score's nonlinearity and a functional interaction part, dominant on DeiT-S and DeiT-B respectively; a hybrid search that ranks with single-block logits and measures only a few candidates needs no dedicated pair measurements and beats the pairwise search in 13 of 14 full ImageNet-1k settings.
Interpretation
The additive assumption behind blockwise distillation search fails under aggressive compression, and the break falls between block-latency budgets of 60% and 55%. Methods such as DNA, DONNA, LANA and Puzzle score each candidate replacement block in isolation and assemble a child by an integer linear program that sums those scores, and this additive premise was rarely tested. On DeiT-S and DeiT-B with seven replacement blocks per layer, seven latency budgets and three fine-tuning seeds per searched network, compared with paired bootstrap confidence intervals: at 55%–40% budgets the additive search trails the pairwise search by 3.3–16.6 top-1 points on ImageNet-100, while at 70%–60% the gap is at most 1.2 points.
The additivity error splits into a part caused by the score's nonlinearity in logit changes and a part caused by genuine non-additive interaction between blocks. Pairwise measurements were previously used as a single lumped interaction term, without separating how much is only curvature of the scoring function. The metric part is computable from cached single-block logits with no extra network evaluations and accounts for 59–65% of the mean KL error on DeiT-S, whereas the functional part accounts for 73–76% on DeiT-B.
A hybrid search first ranks candidate moves by the sum of single-block logits and then measures only the top-ranked few, removing the dedicated pair-measurement stage. The pairwise integer quadratic program needs 1,389 pair measurements, which on 5,000 images amounts to 6.9 million forward images. The hybrid is significantly better than the pairwise search in 7 of 14 ImageNet-100 settings, not significantly different in 6 and significantly worse in 1; on full ImageNet-1k it is significantly better in 13 of 14, at 13–27% of the pairwise stage's forward-image cost.
Perspective
The conclusions are demonstrated on DeiT-S and DeiT-B with seven candidate replacement blocks per layer, at latency budgets from 70% to 40% of the parent's block latency measured on a single RTX 5090; ImageNet-100 children are fine-tuned on 100 classes only, with val1k as a secondary metric, while the full ImageNet-1k experiments fine-tune on 1000 classes and evaluate on the 50,000 official validation images.
Each search is run once per budget, the confidence intervals are conditional on the three fine-tuning runs and are not adjusted for multiple comparisons; the hybrid's cost grows with the number of budgets while the pairwise pair table can be reused across budgets, and at the mean per-budget cost the cumulative costs meet after roughly 26–52 budgets. The stronger-search and fine-tuning-aware scoring studies on ImageNet-100 were not extended to full ImageNet-1k.