Research — ApexSI

This paper is a measured account of pretraining a 21B-parameter mixture-of-experts model on a single GH200, from 21.1% to 50.7% benchmark MFU, with the failed attempts and the released code.

We pretrained a fine-grained MoE model with 20.99B total and 3.36B active parameters on a single NVIDIA GH200. The entire FP32 training state (238.5 GiB) lives in Grace CPU memory, where the GPU’s optimizer kernels update it directly over NVLink-C2C.

Over 13 benchmark measurements, MFU climbed from 21.1% to 50.7%; one of those steps changed the model itself (the expert shape). The gains came from block-level torch.compile, a C++ OpenMP kernel for CPU-side gradient accumulation, Triton fused cross-entropy, keeping FFN gate/up outputs in CPU memory instead of recomputing them, and per-tensor FP8.

In pretraining on real data, per-GPU MFU went from 36.5% to 48.9–49.1% over 10.9B tokens, or an estimated 47.6–47.8% once synchronization is counted. A second GH200 joined through DiLoCo at 4B tokens, and from then on each GPU trained the full model.

Under the 700 W power cap, BF16 dense GEMMs reach only 640–665 TFLOPS (about 66% of peak), which makes it hard to exceed roughly 60% MFU with BF16 alone. We also report what did not work, including Transformer Engine FP8, row-wise FP8, bulk activation offloading, and splitting gradient accumulation between GPU and CPU. MFU is always computed against the BF16 dense peak (989 TFLOPS), even when 47% of the FLOPs run in FP8, so BF16 and FP8 stages can be read on the same scale.

What the paper offers is a measured account of how we combined these techniques: the optimization path, the failures, and where each piece of state lives. Code and logs are released with it, and we hope the record helps teams building LLMs on a small budget.

What we do not claim. We do not propose a new algorithm, and we do not claim a record MFU. Every technique here comes from prior work and is cited where it is used, and every figure comes with its denominator and counting rules. FP8 quality is unchecked, since there is no BF16 control run, and the expert-shape change was not evaluated for model quality.

Key numbers

Benchmark numbers come from synthetic data and real-training numbers from real data. The paper keeps them apart, and so does this page.

Measure	Value	What it means
Benchmark MFUSynthetic data, per GH200	21.1% → 50.7%13 measurements	Each step is one measured change. One of them changed the model itself, the expert shape.
Real-training MFUReal data, per GPU	48.9–49.1%over 10.9B tokens	Up from 36.5% at the start. An estimated 47.6–47.8% once synchronization and checkpointing are counted.
FP32 training stateIn Grace memory	238.5 GiB	All of it sits in Grace CPU memory, updated in place by GPU optimizer kernels over NVLink-C2C.
BF16 dense GEMMAt the 700 W power cap	640–665 TFLOPSabout 66% of peak	With BF16 alone, MFU is hard to push past roughly 60%.
FP8 share of FLOPsFinal configuration	~47%	About 47% of the FLOPs run in FP8, yet MFU is still divided by the BF16 peak.
Document-masking correctionReal-training MFU	−1.6 pointsF × 0.967	Counting attention only within documents lowers real-training MFU by 1.6 percentage points. The headline numbers keep the conventional formula.

MFU is divided by the BF16 dense peak, 989 TFLOPS, even though about 47% of the FLOPs run in FP8 in the final configuration, so BF16 and FP8 stages read on one scale. Against the mixed FP8/BF16 peak of 1,293 TFLOPS, 48.9% would read as about 37%. Benchmark and real-training figures are never mixed.

The MFU ladder

Benchmark MFU after each of 13 measurements. Hover or tap a row to see what changed.

Benchmark MFU per GH200 at 700 W on synthetic data, divided by the BF16 dense peak of 989 TFLOPS. Bars show mfu_at_batch, with the optimizer step amortized over an 8M-token batch. Lighter bars (steps 0–4) were measured with --accum 1, which hides the accumulation cost that later steps include. Step 6 changes the model itself, so read its gain apart from the system-level gains; the paper did not measure model quality. Steps 9 and 10 ran on instance B and the rest on A. Real-training MFU is reported separately, in Figure 1 below.

Figures

Five figures from the paper. Select one to enlarge it and read the full caption.

Diagram of one GH200. On the left, the Hopper GPU holds the 39.1 GiB of BF16 working weights, about 36 GiB of activations and workspace, and all compute. On the right, the Grace CPU holds 238.5 GiB of FP32 state, a BF16 staging buffer, a C++ OpenMP accumulate step and a 72 GiB gate/up storage region. Five numbered flows cross NVLink-C2C, the last being DiLoCo sync with a peer GH200. — **Figure 2.** State placement and data flow on one GH200. The GPU holds the BF16 working weights and does all compute; the Grace CPU holds the 238.5 GiB of FP32 state.

Horizontal bar chart of benchmark MFU for steps 0 to 10, rising from 21.1 percent to 50.7 percent. Bars are coloured by kind: system optimization, model change at step 6, and precision at step 9. Diamonds mark median real-training MFU on instances A and B, and a dashed line marks the BF16 bound at 700 W of 59.7 percent. — **Figure 1.** Benchmark MFU of the full 30-layer model after each change (bars) and median real-training MFU on the two GPUs, A and B (diamonds). The dashed line marks the BF16 ceiling at 700 W (59.7%).

Two panels. Panel a: stacked bars of GPU time by kernel type for one 48K micro-batch, for the BF16 configuration with Triton cross-entropy (2,518 ms, of which dense GEMMs are 1,185 ms and grouped GEMMs 686 ms) and for the torchao row-wise FP8 configuration. Panel b: micro-batch time across six successive configurations, labelled with benchmark MFU from 40.4 percent to 50.7 percent, with a dashed line marking the 59.7 percent MFU ceiling. — **Figure 3.** GPU time for one 48K micro-batch. (a) Breakdown by kernel type. (b) Micro-batch time across successive configurations, against the 700 W ceiling of 59.7% MFU.

Two panels. Panel a: bars showing throughput change against each attempt's own BF16 baseline for seven FP8 approaches; Transformer Engine block scaling, torchao row-wise and the row-wise variants are slower, while tensor-wise linears gain 5.9 percent and tensor-wise with storage gains 7.5 percent. Panel b: benchmark MFU against the activation-memory budget for four configurations, with FP8 tensor-wise and full storage nearly flat at about 50.7 percent. — **Figure 4.** FP8 approaches and combinations. (a) Change in micro-batch throughput for each attempt against its own BF16 baseline. (b) Benchmark MFU against the activation memory budget on instance B.

Two panels. Panel a: benchmark MFU over repeated runs; the PyTorch add_ path and its tuned variants land either in a slow mode near 27 percent or a fast mode near 34 percent, while the C++ OpenMP kernel is always in the fast mode. Panel b: CPU time to add 21 billion elements, from 1.12 seconds for torch add_ to 0.75 seconds for the C++ OpenMP kernel with 32 threads, against a dashed line for GPU compute time of a 32K micro-batch. — **Figure 5.** CPU-side gradient accumulation. (a) Benchmark MFU over repeated runs: the PyTorch add_ path lands in a slow or a fast mode, the C++ OpenMP kernel always in the fast one. (b) CPU time to add 21B elements.

Read the paper

All 20 pages, here in the browser.

Open in new tab Download PDF

Read it in your PDF viewer

The paper is 20 pages. Open it in a new tab, or download it to read offline.

Open in new tab Download PDF

Cite this paper

BibTeX · yoon2026single

@misc{yoon2026single,
  title  = {Pretraining a 21B-Parameter MoE on a Single GH200: A Measured Optimization Path},
  author = {Yoon, Dongwook},
  year   = {2026},
  note   = {Preprint. Code: https://github.com/DW-dev-UE/LLM-from-scratch (commit 1d491e2)}
}

Questions

Is this a new method?

No. The paper proposes no new algorithm and no new technique, and it claims no record MFU. Every technique comes from prior work and is cited where it is used. What it offers is a measured account of how they were combined on a single GH200: the optimization path, the attempts that failed, and where each piece of state lives.

Why divide FP8 runs by the BF16 peak?

So that BF16 and FP8 stages can be read on one scale. MFU here is model FLOPs per token times tokens per second, divided by the BF16 dense peak of 989 TFLOPS, even though about 47% of the FLOPs run in FP8 in the final configuration. Dividing by the mixed FP8/BF16 peak of 1,293 TFLOPS instead would scale MFU by 0.76 and turn 48.9% into about 37%, while BF16 stages would not change.

Can I reproduce it?

Largely, yes. The code is in apex3/ of DW-dev-UE/LLM-from-scratch at commit 1d491e2. The runs used the NGC PyTorch 26.09 container (nvcr.io/nvidia/pytorch:26.09-py3) on a GH200 480GB limited to 700 W. Steps 8 to 10 of the ladder reproduce exactly, step 9 after deleting one loop, while flags can only approximate steps 0 to 7. Two evidence bundles, one per instance, hold the result files behind the numbers, with per-file hashes, and the paper’s appendix lists the environment, flags and commands.