Mixture of experts

1991

Route each token to a few of many expert sub-networks. Jacobs and Hinton 1991, Shazeer 2017, Switch 2021. How frontier models grow parameters without growing cost.

Also: MoE