LLM Bayesian fine-tuning shows state-dependent heavy-tailed gradient noise: geometric-median centering plus residual clipping keeps Fisher-information guarantees on non-log-concave targets
Synopsis
In Bayesian LoRA fine-tuning of LLMs and vision-language models, minibatch gradient errors are heavy-tailed and grow with distance from the final iterate, and a robust estimator combining geometric-median centering with residual clipping, together with adaptive step sizes, retains non-asymptotic Fisher-information guarantees on non-log-concave targets.
Interpretation
Non-log-concave sampling previously required globally Lipschitz gradients and uniformly bounded noise variance, and both assumptions fail even for quadratic potentials or minibatched Bayesian linear regression. Earlier SG-LMC convergence guarantees relied on global smoothness and uniform second moments, and existing extensions did not jointly address BG- noise and gradient-dependent curvature. On a synthetic Gaussian-mixture target, R-LMC tracks the Fisher-information decrease of exact-gradient LMC while SG-LMC diverges; on a radial quartic potential A-LMC reaches lower Fisher information than fixed-step LMC over the observed horizon, and on a radial cubic potential RA-LMC tracks exact-gradient A-LMC.
The method controls estimation error with geometric-median centering and residual clipping, and controls discretization error with a step size that shrinks with the gradient norm. The robust estimator takes the geometric median of three of four independent gradient evaluations as a center and clips the residual of the fourth; the adaptive step scales with a power of the gradient norm, and a noise-aware term shortens it further where oracle noise is large. On OpenBookQA and SLAKE with LoRA fine-tuning of Qwen3-4B and Qwen3-VL-4B-Instruct, RA-LMC outperforms SG-LMC and nine Bayesian adaptation baselines in accuracy and Brier score.
Perspective
The work targets researchers studying non-log-concave sampling and Bayesian fine-tuning, and its theoretical guarantees concern Fisher information of the time-averaged law without assuming dissipativity or a functional inequality. Synthetic experiments cover state-dependent heavy-tailed noise, generalized smoothness, and their combination; LLM and VLM experiments on OpenBookQA and SLAKE support the oracle model and compare predictive metrics.
The theoretical step-size rule requires smoothness and local noise-scale quantities that are difficult to specify for large models, so the practical implementation uses AdamW with Gaussian perturbations. Constants in the complexity bounds depend on problem parameters, and small-accuracy thresholds may depend on them.