Multi-domain fine-tuning stops stepping on itself: 256 examples per domain lift expert routing to 90–98%

Multi-domain adaptation no longer shares one set of low-rank adapters across domains: dense Transformer layers become sparse mixture-of-experts layers, each specialized domain gets a full MLP expert, and the original MLP stays as the general expert. Earlier approaches either trained a separate LoRA adapter per domain and combined them with equal weights, or used a router to weight adapters dynamically while still activating all of them, including irrelevant ones; the low-rank structure also capped adaptation capacity. Across the Llama-3.1-8B and Qwen-3-8B model families, the method reaches the highest biomedical averages (Llama 68.42, Qwen 73.34) and legal averages (Llama 68.96, Qwen 72.15), significantly above the respective second-best baselines (paired t-test), and the highest general-domain averages (Llama 68.82, Qwen 70.08).

The router first learns to tell which domain an input belongs to, so specialized inputs stop calling irrelevant experts and general inputs favor the original MLP. Earlier upcycling methods randomly initialize the router, and domain data alone does not teach it to select the matching expert; after adaptation, a randomly initialized router on Llama rises to 99.62% on general inputs while falling to 51.00% on legal and 42.69% on biomedical inputs. Routers are trained on hidden states from 256 randomly selected examples per domain, and routing accuracy is measured on 128 non-overlapping examples per domain; after adaptation the method reaches 90–98% on specialized domains, against 43–70% for routers trained without this supervision.

Adaptation to each domain updates only that domain's expert and the router while every other parameter stays frozen, so experts that start identical end up moving in different directions. Upcycling baselines that initialize experts by copying the dense MLP still show high cosine similarity between biomedical and legal expert updates after adaptation, 36.83% on Llama and 31.06% on Qwen. Starting from the same copied weights but restricting updates to the matching domain, the method lowers that update similarity to 0.59% on Llama and 0.76% on Qwen, comparable to random-perturbation and partial-reinitialization baselines.

The demonstrated setting is English legal and biomedical adaptation on Llama-3.1-8B and Qwen-3-8B plus a set of general benchmarks, with 393K training examples per domain and only 256 examples per domain for router supervision. An engineer serving several specialized domains on one device can use this to judge whether the method fits their domain count and memory budget.

Inference compute and latency stay bounded by sparse Top-2 activation, but the total static parameter footprint held in GPU memory grows linearly with the number of instantiated domains; the authors list combining the approach with post-adaptation expert quantization or pruning as future work. The one ablation setting that does not improve is the Qwen-3-8B biomedical average, which dips slightly from 73.46 to 73.34.