Apex-3
The next peak of intelligence. A 21B mixture-of-experts model, pretraining in public.
- Status
- Pretraining
- Parameters
- 20.99B total · 3.36B active
- Architecture
- MoE, 64 experts top-4 + 1 shared · 30 layers
- Training
- 1.1T tokens planned, real data
- Context
- 8,192 tokens
- Hardware
- 1× GH200 per GPU worker, 700 W
We’re introducing Apex-3, the first model in our Apex 3 family. It is a 21-billion-parameter mixture-of-experts model, pretrained on real data on a single GH200, in public.
Apex-3 is the largest model we have trained, and the first whose every step is open while it learns. The training code, the raw logs and a 20-page paper on how it runs are already published. The weights will follow on Hugging Face.
Here is what is new in Apex-3:
Architecture. 20.99B parameters, 3.36B active for each token. Thirty layers with 64 routed experts (top-4) and one shared expert, grouped-query attention with 25 sliding-window and 5 global layers, an 8,192-token context and a 204,800-token vocabulary.
Efficiency. The entire FP32 training state, 238.5 GiB, lives in Grace CPU memory, where GPU kernels update it directly over NVLink-C2C. In pretraining on real data, each GPU sustains 48.9–49.1% MFU against the BF16 dense peak at a 700 W power cap, or an estimated 47.6–47.8% once synchronization is counted.
Scale-out. A second GH200 joined through DiLoCo at 4B tokens. Each GPU trains the full model, and the two exchange pseudo-gradients every billion global tokens.
Openness. Everything above can be checked: the code at commit 1d491e2, the step-by-step training logs, and the paper, which also reports what did not work.
Apex-3
The next peak of intelligence.
Benchmark results will be published with the weights. Until then, here is what we can measure without reservation: validation loss on held-out data from each domain of the pretraining mix, at two checkpoints, on both training GPUs. Lower is better.
| Apex-320.0B tokens · GPU A | Apex-320.0B tokens · GPU B | Apex-310.0B tokens · GPU A | Apex-310.0B tokens · GPU B |
|---|
Validation cross-entropy (nats per token) from main_phase1.log on each GPU. GPU B joined at 4B tokens through DiLoCo. Pretraining continues toward 1.1T tokens; Stage 1 ends at 0.51T.
The training run so far
| GPU Afrom 0 tokens | GPU Bjoined at 4B tokens |
|---|
Latest row of each log, snapshot extracted 2026-10-14 20:16 UTC. MFU is per GPU against the BF16 dense peak of 989 TFLOPS, even for FP8 stages; counting attention only within documents lowers it by about 1.6 points.
Availability
Questions
Why validation loss instead of benchmarks?
Apex-3 is still early in pretraining. Downstream benchmarks mean little at this stage, so we will publish them with the weights. Validation loss is what the run itself measures, and it is reported unedited.
How is MFU measured?
Model FLOPs per token times tokens per second, divided by the BF16 dense peak of 989 TFLOPS, even though about 47% of FLOPs run in FP8. Benchmark and real-training figures are kept separate; the paper explains every number.
How does it relate to Apex-2?
Apex-3 is a new model trained from scratch with its own code base. Apex-2, a 3.87B MoE, is our latest released model.