// How to build a 100B model that only costs as much as a 10B model to run
💡 The core idea: Instead of one massive neural network that handles everything, build a team of specialists. A "router" reads each input and picks 2-3 experts most relevant to that token. Math tokens go to math expert. French tokens go to French expert. Only those experts activate — the rest stay off. Bigger capacity, same compute cost.
See which experts activate for different inputs
Pick an input type to see which experts the router activates:
8 EXPERTS (only 2 activate per token)
ROUTER SCORES
Why MoE is efficient
GPT-4 is believed to be a Mixture of Experts model. Total parameter count might be ~1.7 trillion, but only ~200 billion activate per token. This means: a model with 10x more capacity costs only 1x in compute. MoE enables larger, smarter models without proportionally more GPU cost.
ONLY 2-4 EXPERTS ACTIVATE PER TOKENGPT-4, MIXTRAL, GEMINI USE MoE