EntroPrefill couples Rényi-entropy context pruning with conditional stability guarantees, but reports no measured speedup or accuracy result

The paper couples an entropy-guided proposal with explicit deletion constraints: each head is sink-isolated and conditioned, head distributions are pooled within GQA groups using weights exponential in negative collision entropy, and a positive floor parameter limits how strongly entropy may suppress any head. Prior work such as LLMLingua, AttentionRAG, LazyLLM, SlimInfer, and ASL already covers attention-guided compression, intermediate-layer pruning, and adaptive layer selection separately; the paper does not claim novelty for those elements alone, but for coupling the entropy proposal, deletion constraints, independent audit, and conditional output bounds into one fully specified procedure. Stated as propositions on entropy separation and its floor (Proposition 3.1) and on null-calibrated chunk priorities (Proposition 3.2), noting the weights are exponential in negative entropy rather than reciprocals of entropy, and that the floor prevents a suppressed head from receiving zero mixture coverage.

The paper derives a deletion envelope from pooled attention mass to an individual attention row, plus a sharp fixed-row deletion bound for fixed queries and value vectors, and shows the envelope can be attained. It converts the local intuition that small pooled discarded mass is safe into an explicit bound for each constituent proposal row, and exposes the coverage cost arising from mixture weights and distributional discrepancy rather than omitting that cost from the guarantee. Theorem 4.1 obtains the envelope via a pointwise inequality and Cauchy–Schwarz, with a construction where both nontrivial terms equal one; Lemma 4.2 gives a fixed-row bound with an attainable factor of two and explicitly requires fixed queries, retained keys, and values, so it is not a theorem about later layers.

The paper formulates removal as a screening problem with row tolerances and a minimum retained chunk count, and gives a dual upper-bound certificate that requires no strong duality or integrality, separating feasible pruning from unattainable budget targets. The certificate turns the greedy proposal's suboptimality in removed tokens into a computable upper bound, and the paper states explicitly that a greedy proposal failing a target is not itself proof that the target is impossible. Theorem 4.3 holds for arbitrary nonnegative multipliers; the reference proposer sorts by chunk score, breaks ties by decreasing chunk mass then original index, scans once, and its sorting and constraint-update costs are given, with the method positioned as a feasible heuristic rather than an optimizer.

The paper gives a finite-sample simultaneous observer guarantee over all monitored layers and heads using independently drawn audit observers, proves that adaptive layer selection inside the audit event cannot invalidate the established inequality, and adds a conditional first-token perturbation bound with a decision-margin corollary for pre-normalized transformers. Audit samples are drawn independently of the proposal mechanism, so adaptive stopping is not treated as a fixed test; the perturbation bound supplies explicit sufficient Lipschitz constants and is paired with a counterexample showing shallow attention alone cannot yield an unconditional future-output guarantee. Theorem 5.1 builds a simultaneous event from the one-sided Hoeffding inequality and a union bound; Theorem 6.2 and Corollary 6.3 hold under stated domain bounds, and the paper states the result is a first-token statement whose extension to generated sequences needs new bounds and sufficient margins at every step.

The work targets decoder-only transformers with grouped-query attention that delete complete candidate document chunks mid-prefill, in retrieval-augmented generation settings where discarded attention mass is to be held to explicit constraints and adaptive layer selection is to be audited. It also provides resource accounting for composition with paged KV caches and prefill/decode disaggregation, including KV payload formulas for independent per-group allocation versus union materialization in a physical layer block, and an arithmetic break-even condition for pruning. The paper positions the next phase as empirical: testing whether entropy priorities obtain better feasible removals than mass-only or uniform priorities, whether the audit is nonvacuous at practical sample sizes, and whether the conditional stability constants explain observed perturbations, comparing against full-context inference, prompt compression, fixed-layer pruning, and existing methods under matched token and physical-byte budgets.

Several open questions remain for a careful reader: the paper notes the full-model bound can be vacuous because of large Lipschitz products or poor support transfer, and Equation (26) is an assumption about later reference attention rather than an output of the shallow audit, so merely measuring stable chunk ranks does not establish it. The audit controls expected discarded mass under a specified observer distribution, not the worst observer, every document position, or future decode attention, and can be too conservative for short queries. The paper also notes concentrated attention can select a false fact, bridge relevance can emerge only in deeper layers, and irreversible deletion limits recovery during long generation and multi-turn reuse. Because there is no experimental section, whether the audit is nonvacuous at practical sample sizes, how tight the approximations are, and how end-to-end resource trade-offs behave remain unverified.