Training a sparse language model is a balance between cheap inference and costly training. Mixture-of-experts designs shrink what each input costs to run, but the full network still has to live in GPU memory and be refreshed on every pass, and steering tokens to the right expert across a cluster adds its own traffic. Those costs tend to swallow the savings as models grow.
Olmo-core 3, released October 1 by the Allen Institute for AI, is the institute’s attempt to hold that line. It rebuilds the training system behind the Olmo family around mixture-of-experts workloads rather than the dense design Olmo 3 used.
The biggest change is how work is spread across chips. The earlier stack used fully sharded data parallelism, gathering and resharding weights for each small batch. The new one keeps experts resident on GPUs and ships data to them instead. Ai2 measured the result on eight Nvidia B300 GPUs. A 47B-parameter model moved 52,000 tokens per second per GPU. The old stack managed 19,400, putting throughput about 2.7 times higher.
Scaling behavior looks steadier too. Ai2 raised the expert pool from 8 to 128 while still picking four experts per token. Active parameters per token stayed near 3.2B, total capacity grew from 4.6B to 47B, and throughput slipped less than 5 percent. The same infrastructure has been benchmarked past a trillion total parameters. A technical report, open code and a demo accompany the release.
