Query-conditioned distributional hypernetworks predict LoRA weight updates, where the distribution mean alone beats deterministic hypernetworks and weight sampling enables a new form of test-time scaling
This work studies hypernetworks conditioned only on an LLM's input query to estimate LoRA, and introduces distributional hypernetworks that output a distribution over LoRAs via an end-to-end loss with a differentiable Monte Carlo approximation; results show that even the mean of the learned distribution can outperform deterministic hypernetworks, that sampling weight updates yields a form of test-time scaling distinct from token sampling with performance improving as more weight samples are considered and remaining stronger than corresponding token-sampling adaptation baselines, and that generated updates can transfer across queries.