Robust aggregators scrub federated watermarks: camouflaging watermarked updates as inliers keeps one client's 3.3% effective watermark detectable

When a federated server aggregates client updates with a strong Byzantine-robust aggregator, existing LLM text watermarks are removed during training as anomalous updates, and black-box detection after training fails. In centralized fine-tuning, models trained on watermarked data remain detectable with high confidence even after further fine-tuning on clean data; this work moves that detection setting into federated learning and has the server actively filter updates with RandEigen during training. Across Pythia, Qwen, and LLaMA on Alpaca and OpenWebText, four representative watermarks (KGW+, AAR, KTH-EXP, KTH-ITS) are mostly detectable under FedAvg, while under RandEigen most p-values return to around 0.50; the only exception is LLaMA on Alpaca with KTH-EXP at 0.069, still above the 0.05 threshold.

Treating the continuous input embeddings of watermarked samples as adjustable variables, so that the gradients they induce stay close to the clean-update gradient distribution while the embeddings remain near the original token sequence, keeps watermarked updates inside the aggregator's acceptance region while the watermark signal stays in the text. Simply scaling down watermarked updates or mixing clean and watermarked updates with a single convex coefficient only moves the update direction along one line segment and cannot correct coordinate-wise outlier structure; this work instead solves a constrained optimization in embedding space, using a reconstruction loss to match clean proxy gradients and a cosine-distance regularizer to stay near the original embeddings. Clean gradients are estimated by a local proxy model that each watermarking client keeps fine-tuning on its own clean data; on Qwen fine-tuned on Alpaca, CamoMark reaches p-values down to roughly the 10^-4 range, whereas the convex-combination baseline succeeds only for Qwen on Alpaca and is unstable in every other configuration.

Under post-training removal attacks including continued fine-tuning, pruning, and distillation, the watermark remains detectable in most settings, and watermarking has little effect on model utility. At an effective watermark ratio of 3.3%, Pythia is the most sensitive to removal attacks, and LLaMA loses detectability on OpenWebText under longer continued fine-tuning and stronger pruning; raising the effective watermark ratio to 6.6% leads the authors to report that every model, dataset, and attack combination stays detectable. Utility comparisons show CamoMark under RandEigen raises C4 perplexity in about half the settings and lowers it in the other half relative to baseline, with a small maximum increase that the authors attribute to RandEigen's over-filtering rather than to CamoMark itself.

The work targets data provenance in federated LLM fine-tuning: a data contributor holds its own watermarked data and private key and, after training, verifies through black-box access alone whether a suspect model was trained on that data. Experiments cover the Pythia, Qwen, and LLaMA model families on Alpaca and OpenWebText, with RandEigen as the main server aggregator; the authors add a sanity check of CamoMark under Filtering and No-Regret on Pythia trained on Alpaca and describe the design as aggregator-agnostic. They also evaluate non-IID client data using nearest-neighbor mixing followed by Multi-Krum, and note that strong robust aggregators remain underexplored for non-IID settings.

Detection is reported as statistical evidence via p-values, which the authors explicitly do not treat as definitive proof of ownership or legal violation. At an effective watermark ratio of 3.3%, Pythia loses detectability after sufficiently long continued fine-tuning, after continued fine-tuning on a different dataset, and under some pruning ratios, and LLaMA loses detectability on OpenWebText under longer continued fine-tuning and stronger pruning. The authors also note that there is currently no theoretical characterization of how much residual watermark signal guarantees successful detection after training, so detectability is assessed empirically.