One training step's opened values are enough to copy a LoRA adapter: VeilTune moves the forward pass, backpropagation and Adam update into secure multi-party computation
Synopsis
Existing secure fine-tuning protocols hand the data owner hidden representations or logits from every step, and from those the data owner can copy the provider's LoRA adapter: from the hidden representations of a single mini-batch of 32 samples, the copy agrees with the original on 99.2% of test inputs, and logits alone give about 94%. VeilTune instead runs the forward pass, backpropagation and the Adam update entirely in secure multi-party computation and reveals only the trained adapter to the model owner, averaging 5.3 points above SecP-Tuning on four few-shot tasks with RoBERTa-large and staying within 0.3 to 1.7 points of plaintext training.
Interpretation
In secure fine-tuning, the intermediate results the data owner receives are by themselves enough to copy the provider's adapter, even when the data owner follows the protocol. Earlier secure fine-tuning protocols protected the data owner's samples but opened the hidden representations or logits of every training step to the data owner, on the grounds that these values are not the adapter itself. On SST-2, starting from the hidden representations of a single mini-batch of 32 samples, the extracted adapter agrees with the original on 99.2% of test inputs, and logits alone give about 94%.
VeilTune runs the entire training step inside secure multi-party computation and reveals only the finished adapter, to the model owner. SAP and PrivTuner backpropagate from values opened to the data owner, and SecP-Tuning avoids backpropagation altogether, while forward-only zeroth-order training needs about 16 times more steps and 24 times more communication to converge. On four few-shot tasks with RoBERTa-large and 16 training examples per class, VeilTune stays within 0.3 to 1.7 points of plaintext training and averages 5.3 points above SecP-Tuning.
Three techniques make backpropagation in secure multi-party computation practical: token packing computes only the positions the loss depends on, training-aware nonlinear operators stay accurate over a wide input range, and oblivious step skipping discards updates corrupted by truncation failures. Generic secure operators are accurate only on the typical range of inference activations, whereas training produces LayerNorm variances above 100 and Adam second moments at magnitudes that generic operators and the fixed-point encoding represent poorly, and such an error enters the optimizer state and distorts every later update. On SST-2, VeilTune takes 2.5 hours and 3.9 TB of communication per party end to end, against 8.8 hours and 14.2 TB for SecP-Tuning and 65.9 hours and 67.4 TB for full-parameter secure fine-tuning.
Perspective
The demonstrated setting is LoRA fine-tuning of RoBERTa-large on four few-shot classification tasks, SST-2, Yelp, AG News and RTE, with 16 training examples per class, a rank-4 adapter on the query and value projections of the last eight blocks, two parties each on an A100, and timing over emulated LAN and WAN links.
The security argument covers a static, semi-honest adversary that corrupts one party, and does not cover protocol-deviating attacks or side channels; the share of skipped steps grows with input length, about 10% on SST-2, about 40% on AG News and RTE, and about half on Yelp, and the accuracy gap comes from those skipped steps.