Models

The Apex family

Four models, all trained from scratch. Three released, one pretraining in public.

Released: Apex-2 · Apex-2-Base · Apex-1-DPO
In training: Apex-3
Largest: Apex-3 · 20.99B MoE, 3.36B active
Latest release: Apex-2 · 3.87B MoE, 1.45B active
Weights: huggingface.co/YOON1v

Every Apex model is trained from scratch, without external pretrained weights. Each release comes with benchmark results we measured ourselves, reported as they are, including where the model falls short.

Apex-320.99B MoE · 3.36B active

A fine-grained mixture-of-experts model, pretraining on real data on a single GH200.

In training

Introducing Apex-3 Training log

Apex-23.87B MoE · 1.45B active

Our latest release: pretrained on 86.5B tokens and fine-tuned for chat.

Available

Introducing Apex-2 Hugging Face

Apex-2-Base3.87B MoE · 1.45B active

The pretrained model behind Apex-2, before SFT. For your own post-training.

Available

Introducing Apex-2-Base Hugging Face

Apex-1-DPO1.12B dense

Where our lineage began: 20B tokens of pretraining, then SFT and DPO.

Available

Introducing Apex-1-DPO Hugging Face

Side by side, the family grows from a 1.1B dense model to a 21B mixture of experts, each trained on more data than the last.

	Apex-3in training	Apex-2SFT	Apex-2-Basepretrained	Apex-1-DPOSFT + DPO
ParametersTotal	20.99B	3.87B	3.87B	1.12B
ParametersActive per token	3.36B	1.45B	1.45B	1.12B
ArchitectureFFN	MoE64 routed (top-4) + 1 shared	MoE16 experts, top-4	MoE16 experts, top-4	DenseSwiGLU
ArchitectureLayers	30	32	32	24
ContextTokens	8,192	4,096	4,096	4,096trained at 2,048
VocabularyTokenizer	204,800SuperBPE	151,936Qwen3 tokenizer	151,936Qwen3 tokenizer	32,000byte-level BPE
PretrainingTokens	1.1Tplanned	86.5B	86.5B	~20B
Post-trainingStages	—	2-stage SFT	None	SFT, DPO
Loads withHugging Face	At release	transformers, vLLM	transformers, vLLM	transformers

Benchmarks across the family

	Apex-2SFT	Apex-2-Basepretrained	Apex-1-DPOSFT + DPO
CodeHumanEval	43.9	36.6	8.5
CodeMBPP	56.3	54.8	5.2
MathGSM8K	32.40-shot CoT	15.18-shot	1.95-shot
CommonsenseHellaSwag	62.2	60.8	46.9
CommonsenseARC-e	64.1	66.7	52.6
CommonsenseARC-c	38.4	39.7	30.7
CommonsensePIQA	74.8	73.9	68.6
CommonsenseWinoGrande	61.8	60.1	52.8

Apex-2 and Apex-2-Base code scores use EvalPlus; Apex-1-DPO was measured with lm-evaluation-harness on the TinyLlama protocol, so code and math rows are not strictly comparable across generations. Apex-3 will be benchmarked at release; its validation loss is on the Apex-3 page. Full tables are on each model’s page.

Quick start

generate.py

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOON1v/Apex-2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto")

ids = tok("def is_prime(n):", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128)
print(tok.decode(out[0], skip_special_tokens=True))

Plain text continuation works for every model. For chat, see each model card for its chat template.

Which model should I use?

For chat, code and instructions

Apex-2. It is our strongest released model and runs with transformers and vLLM.

For my own fine-tuning

Apex-2-Base, the same model before SFT.

For small, fast experiments

Apex-1-DPO, a 1.1B dense model under Apache 2.0.

For the largest model

Apex-3 is still pretraining. Follow its training log; the weights will follow on Hugging Face.