Skip to main content

# QLoRA fine-tuning

2 related articles

Subscribe to updates

arXiv Nov 13, 2026
Clinical small models learn when to stop: PECA cuts mean answers 72.8% and lifts MedMCQA accuracy to 44.0%
Small language models for clinical decision support often answer at far greater length than the question requires after fine-tuning, and existing length control relies on preference data, reward models or proprietary teachers. PECA adds a differentiable conciseness term to the QLoRA supervised fine-tuning objective: once generation passes a token budget, it raises the end-of-sequence probability, with no preference data, no auxiliary model and a single commodity GPU. On the 1.7-billion-parameter Qwen3-1.7B-Instruct, PECA reduces mean response length from 256.0 to 69.7 tokens, a 72.8% cut, while raising MedMCQA multiple-choice diagnostic accuracy from 40.8% to 44.0%.
Small language models for clinical decision support often answer at far greater length than the question requires after fine-tuning, and existing length control relies on preference data, reward models or proprietary teachers. PECA adds a differentiable conciseness term to the QLoRA supervised fine-tuning objective: once generation passes a token budget, it raises the end-of-sequence probability, with no preference data, no auxiliary model and a single commodity GPU. On the 1.7-billion-parameter Qwen3-1.7B-Instruct, PECA reduces mean response length from 256.0 to 69.7 tokens, a 72.8% cut, while raising MedMCQA multiple-choice diagnostic accuracy from 40.8% to 44.0%.
Tanzeel Abbas（First author）
Parameter-Efficient Conciseness Adaptation QLoRA fine-tuning Medical Question Answering Small language models
arXiv Oct 14, 2026
A 4B clinical scribe plus a fixed verifier keeps three quarters of notes on-device and matches frontier acceptance
Clinical dialogue-to-note generation is high-volume and intolerant of fabrication; a 4B-parameter model inside a fixed four-evaluator verification stack comes close to a frontier model on factual entailment, with its deficit concentrated in inter-section structure. Treating verifier rejection as an escalation signal reaches 99.6% end-to-end acceptance against 98.4% for frontier-only, statistically indistinguishable, while three quarters of encounters need no frontier generation call.
Clinical dialogue-to-note generation is high-volume and intolerant of fabrication; a 4B-parameter model inside a fixed four-evaluator verification stack comes close to a frontier model on factual entailment, with its deficit concentrated in inter-section structure. Treating verifier rejection as an escalation signal reaches 99.6% end-to-end acceptance against 98.4% for frontier-only, statistically indistinguishable, while three quarters of encounters need no frontier generation call.
Nina Fatehi（First author）
Small language models Verification stack Clinical Documentation QLoRA fine-tuning

You’re all caught up