Mingzi Cao

Dense Transformer layers are expanded into sparse mixture-of-experts layers, giving law and biomedicine their own full MLP experts while a router is supervised with a small number of examples to pick the right expert per domain. On Llama-3.1-8B and Qwen-3-8B the method posts the highest biomedical and legal averages and the highest general-domain average; with only 256 examples per domain for router supervision, routing accuracy on specialized domains reaches 90–98% after adaptation, against 43–70% for routers trained without it.

Dense Transformer layers are expanded into sparse mixture-of-experts layers, giving law and biomedicine their own full MLP experts while a router is supervised with a small number of examples to pick the right expert per domain. On Llama-3.1-8B and Qwen-3-8B the method posts the highest biomedical and legal averages and the highest general-domain average; with only 256 examples per domain for router supervision, routing accuracy on specialized domains reaches 90–98% after adaptation, against 43–70% for routers trained without it.