Mixture of Experts · MoE
Mixture of Experts
What is Mixture of Experts?
- Activates only the experts needed, out of many
- Big, yet saves compute (sparse activation)
- Adopted by the latest large models
What MoE really changes is the link that tied model size and inference cost together. Until now, doubling the parameters roughly doubled the compute needed to process each token. With MoE the two are decoupled, so when you read a model spec, check both numbers: total parameters and active parameters. Total size lifts answer quality, while active size determines the amount on your bill.
Structurally, the common design splits the feedforward layer of a transformer block into several experts, and a router picks just the top one or two for each token. With 64 experts and only 2 in use, about 3% are active at any time. Don’t confuse this with an ensemble. An ensemble runs every model and merges the results, but MoE is built to run only the one it picks. Also, experts are not divided into fields people can recognize, such as “legal” or “medical.” They split apart on their own during training, so no one can assign which expert handles what, and training adds a separate load-balancing loss to keep any single expert from being overused.
The criteria for adopting it are surprisingly simple. If your company rents models through an API, MoE shows up only in token pricing and response speed, so there is little payoff in examining the internal structure. If you plan to host and run a model yourself, though, it’s a completely different story. Compared with a dense model of the same quality, it’s a trade of more memory for less compute, and that changes the criteria for choosing hardware.
Memory is where things most often break in practice. Teams calculate that “with 13 billion active parameters, one GPU should do,” start building in-house, and then hit the wall: all the total parameters must be loaded into memory, and the hardware quote jumps severalfold. They missed that MoE reduces computation, not memory. The same goes when concurrent requests pile up. If the tokens in a batch each call different experts, most experts end up awake anyway, and the savings you expected vanish first during peak traffic hours.
Miru LeeExpert in future careers, self-branding, and AI tools training
Kwangdong MinExpert in business models, revenue models, and market researchThe mixture-of-experts concept was proposed in 1991 in work by Jacobs, Hinton, and others, and it has regained attention in the large language model era as an efficient way to scale.
Consider a hypothetical case. Company A, a 240-employee claims adjusting contractor for property and casualty insurers, plugged a large language model API into its customer inquiry handling last year. As monthly calls grew to 3.8 million, the bill hit KRW 21 (≈ US$1) million, and management declared, “Next year’s AI budget is half of this year’s.” That left a three-month assignment: cut costs without cutting quality.
- When Team Lead Park of the AI Platform Team classified 30,000 inquiry logs, 62% were concentrated in just two types: “insurance payout criteria” and “document resubmission,” while the rest were scattered in a long tail.
- Team Lead Park proposed switching to an MoE model with 141 billion total parameters and 13 billion active parameters, and spent two weeks verifying the vendor’s claim of “the same quality at one-third the per-token cost” against 5,000 real inquiries.
- Kim, the infrastructure manager who was weighing an on-premises option, brought an H100 8-GPU quote and said, “Even with 13 billion active, the GPUs have to load all 141 billion at once.” The team dropped the in-house build and stayed with the API.
- On the day a batch of overnight settlement inquiries was pushed through all at once, response latency spiked to 1.7 times normal. The team added a queue that feeds requests through at 3,000 per hour and diverted simple, repetitive inquiries to a cache.
- Three months later, the monthly API bill fell from KRW 21 million (≈ US$15,700) to KRW 7.8 million (≈ US$5,840), and the average first response time for agents dropped from 4 minutes 20 seconds to 2 minutes 50 seconds.