Delta-Matching Reports FP8 LLM Training Parity Through 5.29B Parameters
Published by AINave Editorial
MIT, Carnegie Mellon University and NVIDIA Research researchers report that Delta-Matching matched BF16/FP32 mixed-precision results in tests up to 5.29 billion parameters. The method targets a specific failure in FP8 attention training: mismatched scaling between the forward pass and the backward pass. The result is a research finding, not evidence yet that the approach works at frontier-model scale or in production.
The failure emerges in attention backpropagation
FP8 training can quantize attention values using one scaling factor in the forward pass. But memory-efficient attention may recompute intermediate values during backpropagation, where the calculation can effectively assume a different factor. The Tech Times account of the research calls this a “stale delta” mismatch. It says the mismatch breaks a zero-row-sum property of the softmax Jacobian, biasing gradients in a way that can accumulate during training. The article describes the resulting gap growing at larger tested scales.
The reported comparisons show why a fix matters. At 1.67 billion parameters, naive FP8 had validation cross-entropy of 1.8970, compared with 1.4178 for BF16. At 5.29 billion parameters, the stale-delta hybrid scored 16.3% on RULER-8K, against 53.5% for the BF16 baseline. Those are different tests and metrics, so they should not be read as a single scorecard.
| Method and test | Reported result |
|---|---|
| Naive FP8, 1.67B parameters, validation cross-entropy | 1.8970 |
| BF16, 1.67B parameters, validation cross-entropy | 1.4178 |
| Stale-delta hybrid, 5.29B parameters, RULER-8K | 16.3% |
| BF16, 5.29B parameters, RULER-8K | 53.5% |
The correction preserves FP8 through attention
Delta-Matching adjusts scaling in block-scaled FP8 matrix multiplications to keep the forward and backward calculations consistent and restore the softmax gradient invariant. The account says the method reconciles E4M3 scaling in the forward pass with E5M2 scaling in the backward pass, allowing attention-core operations to run natively in FP8 in the tested setup. It reports parity with the BF16/FP32 mixed-precision baseline across 569 million, 1.67 billion and 5.29 billion parameters. The reported results cover those scales, not larger frontier models.
That distinction matters for the efficiency claim. The article cites H100 peak FP8 throughput of 3,958 teraflops with sparsity, versus 1,979 for BF16. Those hardware figures suggest headroom, but they do not establish an end-to-end training speedup or cost reduction; realized gains depend on how the method performs in a full workload.
The researchers describe the correction as requiring no architecture change or reduced global batch size. But implementation materials, including code, checkpoints and data recipes, are described as forthcoming, and the evidence supplied does not establish independent replication or production deployment. Until results reach tens or hundreds of billions of parameters, Delta-Matching is best understood as a promising way to remove a specific numerical obstacle, not a guarantee that FP8 can replace mixed precision across large training runs.
