AoE-MTL: Gate-Free Expert Selection for Efficient Multi-Task Learning

Multi-gate Mixture-of-Experts (MMoE) architectures use task-specific gating networks for expert selection, but these gates add parameters, tuning overhead, and can suffer from expert imbalance. We propose AoE-MTL, a gate-free expert architecture where experts self-activate using task-specific L2-norms of their outputs. Built on a CGC-style mix of shared and task-specific experts, AoE-MTL enables flexible task routing with zero gating parameters or auxiliary balancing losses. Because activation is computed directly from expert outputs, routing reflects each expert’s actual contribution for a given task. Across NYUv2, PASCAL-Context, Cityscapes, and Office-Home, AoE-MTL matches or surpasses MMoE, CGC, and PLE, achieving average improvements of 9.31%, 2.09%, 5.23%, and 14.21% over MMoE with fewer parameters. These results show that autonomous norm-based expert selection is a simple, stable, and effective alternative to explicit gating in multi-task learning.