AMD Instella-MoE-16B-A3B Opens Up Sparse LLM Research
Published by AINave Editorial • Reviewed by Ramit
AMD has released Instella-MoE-16B-A3B, a decoder-only MoE model with 16B total parameters and 2.8B active parameters per token. The practical takeaway for AI builders is straightforward: AMD is exposing more of the training and serving stack than a typical model release, but the weights are intended for academic and research use rather than unrestricted commercial deployment.
What AMD released
Instella-MoE-16B-A3B was trained from scratch on AMD Instinct MI300X and MI325X GPUs. AMD is publishing weights from each training stage, data mixtures, training configurations, and inference code under ResearchRAIL. The training codebase is MIT licensed, creating an important distinction: the tooling is broadly reusable, while the model weights have tighter usage conditions.
The model uses 27 layers, a hidden size of 2,048, 16 attention heads, and a 128,896-token vocabulary. Each MoE layer has two shared experts and selects six routed experts from a pool of 64. That routing design lets the model retain 16B total parameters while using a much smaller active parameter set for each token.
Why the sparse design matters for builders
A mixture-of-experts LLM does not run every feed-forward expert for every token. A router selects a subset, which can reduce the computation required for each forward pass compared with a similarly sized dense model. It does not eliminate deployment complexity, though. Expert routing introduces communication and memory-management challenges, especially when experts are distributed across multiple accelerators.
AMD highlights two systems-level choices. Gated Multi-head Latent Attention adds an input-conditioned output gate to the attention mechanism. FarSkip-Collective is designed to overlap expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to a 39.2% reduction in time to first token during expert-parallel serving, although these are vendor-reported results tied to particular hardware and evaluation setups.
For teams studying MoE latency, routing, or accelerator-aware inference, the value is the visibility into those design choices. This is more useful than simply receiving a checkpoint with no explanation of how it was trained.
Long context and training details are part of the release
Pre-training used 7.1T tokens drawn from Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360. Mid-training used three variants of Dolma3 Dolmino 100B. Later stages included supervised fine-tuning, direct preference optimization, and reinforcement learning for instruction following.
The model’s context window was extended from 4K to 64K tokens using YaRN, a higher RoPE theta, and document masking. That makes Instella-MoE-16B-A3B relevant to long-context MoE experiments, but a nominal 64K window is not the same as reliable performance across every long-document workload. AMD reports long-context results of 41.5 on HELMET and 79.4 on RULER in the supplied evaluation summary.
Benchmark results need careful reading
The base model is reported at 76.7 on the cited aggregate comparison, ahead of Moonlight-16B-A3B at 76.2 and other listed fully open models. It also reports 86.5 on WinoGrande and 65.7 on HumanEval+. At the same time, it trails Qwen3.5-4B-Base at 79.5 in that comparison.
Those numbers make Instella-Mo
