Ayush K Tarun

COAX adds a positive routing bias toward a designated expert subset for each data category during pretraining, letting languages, modalities, or domains with tiny data shares claim a larger expert share; across MoE models from 345M to 10.6B, low-resource categories lose less, while inference stays standard top-k routing.

COAX adds a positive routing bias toward a designated expert subset for each data category during pretraining, letting languages, modalities, or domains with tiny data shares claim a larger expert share; across MoE models from 345M to 10.6B, low-resource categories lose less, while inference stays standard top-k routing.