Every step, on the record.
Loss, throughput and validation, exactly as the training runs wrote them.
Questions
How is MFU measured?
MFU is model FLOPs per token times tokens per second, divided by the BF16 dense peak of one GPU, 989 TFLOPS. The peak stays BF16 even for FP8 stages, where about 47% of the FLOPs run in FP8. The FLOP count treats each row as one document; counting attention only within documents lowers real-training MFU by about 1.6 points (×0.967). The paper explains every number.
Why two GPUs?
Apex-3 trains on one GH200 per GPU worker. GPU A trained alone until 4B tokens, when a second GH200, GPU B, joined through DiLoCo. Each GPU trains the full model, and the two exchange pseudo-gradients every billion global tokens. That is why GPU B’s curves start at 4B tokens, and why the run events list every sync.
Where are the logs for Apex-2?
They are published with the weights on Hugging Face, in each model’s repository. Pick Apex-2 or Apex-2-Base above for the link to its files, or read the model pages: Apex-2 and Apex-2-Base.