VALSE proposes a per-sample non-contiguous layer-skipping framework with an MoE duality proof, but prototype routing collapse leaves its core hypothesis unverified

It gives a closed-form expected-FLOPs expression for arbitrary per-sample skip schedules and proves that the skip-layer function space is strictly contained in the full-layer function space, with an explicit separating example. Prior adaptive-depth work lacked a computational-cost characterization for arbitrary (including non-contiguous) skip schedules and did not characterize the function-space relation between skip-layer and full-layer models. Theorems 2, 10 and 11 are fully proven, and theorems 10 and 11 hold only under identity residual skipping (Scheme A); the prototype does not directly test these theoretical results.

It proposes VALSE: five interpretable features extracted from the first few layers (hidden-state norm, inter-layer delta norm, self-attention entropy, prediction entropy, singular-value concentration) yield a per-sample difficulty score, which feeds three routing variants (hard, soft, global) producing non-contiguous retention sets, supported by four auxiliary losses and a three-stage curriculum. Existing methods are either truncation-based early exit (prefix-only retention), per-token routing (no sample-level difficulty semantics), or static pruning; VALSE extends conditional computation from the width to the depth dimension while retaining arbitrary middle layers. The method and training scheme are specified in full, but the prototype omits curriculum Stage 1 and uses a fixed Gumbel temperature, so the design is not fully executed.

It establishes a structural duality between MoE and VALSE: both share a router-selection-sparse-execution-load-balancing template, acting on the orthogonal width and depth dimensions, and can be composed into two-dimensional sparse activation. No prior work unified horizontal MoE and vertical layer skipping into a single sparse-selection framework, leaving the two-dimensional sparse-activation regime unoccupied. Proposition 6 provides an isomorphism proof and theorem 9 a unified mask representation; the text states this is a structural rather than functional duality, and the hybrid FLOPs advantage (proposition 8) is theory-only and unverified.

The prototype evaluation reports a transparent negative result: on SST-2 the 12-layer model's average activated layers dropped from 12 to 3 (62.1% theoretical FLOPs savings) at accuracy 0.62 versus 0.63 for full depth, but this stems from routing collapse to minimum depth, and the Pearson correlation between difficulty and activated layers was negative, opposite to the hypothesis. The result turns routing collapse from a theoretical risk (proposition 17) into an observed phenomenon and offers a collapse-equilibrium diagnosis via proposition 21 with three remediation directions. Single CPU machine, single dataset, single random seed, 300 training samples, no external baselines; the authors explicitly frame this as a prototype-scale negative result rather than a performance benchmark.

The work addresses readers working on adaptive-depth conditional computation and sparse Transformers, especially those interested in extending sparsity from width to depth. The theoretical portion (expected-FLOPs formula, function-space containment, MoE duality) is self-contained under its stated assumptions and can serve directly as an analytical tool for follow-up work; the method portion provides a reproducible blueprint through its design contract (five features, three routing variants, four auxiliary losses, three-stage curriculum). The authors state clearly that the scope of the conclusions is prototype scale: 12 layers, roughly a million parameters, trained from scratch, on the single SST-2 dataset, and that the core difficulty-adaptive routing hypothesis was not verified at that scale. Testing whether the method holds requires re-running at larger scale with pre-trained weights, multiple datasets and multiple seeds, and completing external baselines and ablations.

Several questions remain open. First, the difficulty-depth monotonicity (proposition 4) is a population-level statement resting on the assumption that marginal layer contribution is non-decreasing in difficulty, and its empirical validity on large pre-trained models is not established by the prototype. Second, proposition 21 characterizes a degenerate routing-collapse equilibrium but does not prescribe a hyperparameter regime that provably avoids it; the three remediations proposed (reduce the balance-loss weight, lower and anneal the gate bias, restore the three-stage curriculum) are falsifiable hypotheses derived from that proposition and have not been executed. Third, the batch-internal min-max normalization makes the difficulty score a within-batch rank statistic that is not comparable across batches, and the EMA-calibrated variant described is a production-target configuration rather than a run result. Fourth, the extensibility of non-contiguous skipping to autoregressive generation (position-agnostic gate, causal-mask compatibility, self-speculative decoding fallback) is an architectural design property that has not been empirically tested. Fifth, theoretical FLOPs savings are not converted into wall-clock latency reduction, leaving engineering work outstanding. In addition, although this is the full text, some tables and hyperparameter values appear as placeholders in the body, and exact numbers should be taken from the training log and code to be released with the paper.