The expert panel analogy

Imagine a hospital with eight specialist doctors: a cardiologist, a neurologist, a dermatologist, and so on. A patient walks in. You do not make all eight examine every patient, that would be slow and wasteful. Instead, a receptionist glances at the symptoms and routes the patient to the two most relevant specialists.

That is exactly how Mixtral works. The "specialists" are small sub-networks called experts. The "receptionist" is a small, fast component called the router.

flowchart TD
    Token[A word comes in] --> Router{Router picks the<br/>2 best experts<br/>out of 8}
    Router --> E1[Expert 1]
    Router --> E3[Expert 3]
    E1 --> Combine[Combine the<br/>two experts' work]
    E3 --> Combine
    Combine --> Out[Output for this word]
    Router -. not used this time .-> Skip[Experts 2, 4, 5, 6, 7, 8<br/>stay asleep]