AMD wants to be known for the blueprint as much as the model. Its newest open release, Instella-MoE-16B-A3B, is a mixture-of-experts LLM trained from scratch on Instinct MI300X and MI325X GPUs that reaches 16B parameters while activating just 2.8B per token. Weights carry a research-only license, but the training code ships under MIT and the full pipeline is documented at every stage.
Evaluations put the base checkpoint at 76.7 on average, which leads the fully open field: Moonlight-16B-A3B posts 76.2 and OLMo-3-7B posts 70.1, while Qwen3.5-4B-Base stays ahead at 79.5. The tuned version climbs from 71.58 to 73.22 after supervised fine-tuning and reinforcement learning.
Two engineering moves define the release. Gated Multi-head Latent Attention places a learned gate on the attention output, and FarSkip-Collective feeds stale and partial activations into the MoE and attention layers so expert-parallel communication overlaps computation. AMD measures a 12.7% pre-training speedup and up to 39.2% faster time to first token.
The training run consumed 7.1T tokens drawn from open sources such as Nemotron-CC-v2, MegaMath and RefineCode, with a long-context phase stretching the window to 64K through YaRN. BF16 weights need about 32 GB of memory, so one high-memory accelerator can host the model, and SGLang inference code is included.
Researchers get a complete end-to-end MoE recipe. Commercial teams will have to wait for licensing terms that allow production use.
