Federated LLM aggregation stops trusting pairwise similarity: LoRA updates become hypergraph nodes, and malicious weights drop to near zero

Federated LLM aggregation gains a self-healing step: the server first judges which agents' updates are semantically inconsistent, then sets aggregation weights, instead of averaging every uploaded update equally. Earlier Byzantine defenses targeted federated learning in general, using robust aggregation, update similarity or secure computation to filter malicious contributions; how malicious updates disrupt semantic consistency in federated LLMs had not been specifically addressed. On AG News, HMP-GAE reaches 88.95% accuracy with Qwen-2.5, exceeding Multi-Krum, FoolsGold and FLTrust by 3.27%, 0.95% and 1.48%, and 91.44% with Llama-3.2, exceeding them by 2.28%, 1.81% and 1.21%.

Each agent's update is first represented as a node in a hypergraph, and hyperedges connect several updates at once; the server passes messages back and forth between nodes and hyperedges to obtain embeddings that reflect group relations. Hypergraph message passing, which iteratively exchanges information between nodes and hyperedges, replaces graph structures that only see pairwise correlations, so it can capture high-order relations that single-update or pairwise comparisons may miss. A graph autoencoder reconstructs node connectivity and hyperedge membership, and its reconstruction error together with consistency against historical checkpoints sets each agent's aggregation weight; the two malicious weights fall to approximately zero while five benign users keep stable positive weights.

An ablation removes hypergraph message passing and hyperedge reconstruction separately to test each component. Without hypergraph message passing and with only the graph autoencoder, Qwen-2.5 on Yahoo! Answers drops from 61.12% to 59.73% accuracy, while semantic entropy rises from 0.6378 to 1.1034 and perplexity from 1061.74 to 1220.49. Removing hyperedge reconstruction in the same setting gives 60.01% accuracy, 0.7449 semantic entropy and 1130.91 perplexity, indicating that both components contribute to the final metrics.

Evaluation covers two text classification datasets, AG News and Yahoo! Answers, with Qwen-2.5 and Llama-3.2 backbones, and the server uses only aggregated parameters and historical checkpoints without touching private local data. Per-user trust weight analysis shows HMP-GAE drives two malicious users' weights to approximately zero and keeps them suppressed in later rounds while assigning stable positive weights to five benign users. A reader interested in non-classification generation tasks or larger agent counts would need validation in those settings.

Per-class perplexity shows HMP-GAE is lowest across all four AG News classes, but not in every Yahoo! Answers class: with Qwen-2.5, FoolsGold is lower on Science & Mathematics and FLTrust is lower on Health, Sports and more clearly on Politics & Government; with Llama-3.2, FLTrust is lower on Sports and Politics & Government. Assigning trust at the user level can reduce the contribution of an untrustworthy user who still supplies useful information for a particular class, creating a trade-off between suppressing malicious contributions and retaining class-specific knowledge.