LLM Bayesian fine-tuning shows state-dependent heavy-tailed gradient noise: geometric-median centering plus residual clipping keeps Fisher-information guarantees on non-log-concave targets
In Bayesian LoRA fine-tuning of LLMs and vision-language models, minibatch gradient errors are heavy-tailed and grow with distance from the final iterate, and a robust estimator combining geometric-median centering with residual clipping, together with adaptive step sizes, retains non-asymptotic Fisher-information guarantees on non-log-concave targets.