Introducing

Apex-2-Base

The pretrained model behind Apex-2. Before any fine-tuning, ready for yours.

Repository: YOON1v/Apex-2-Base
Parameters: 3.87B total · 1.45B active
Architecture: MoE, 16 experts, top-4 · 32 layers
Training: 86.5B pretraining tokens, no SFT
Context: 4,096 tokens
Runs with: transformers · vLLM

Apex-2-Base is the pretrained model behind Apex-2: a 3.87B mixture-of-experts with 1.45B active parameters, after 86.5B tokens of pretraining and before any fine-tuning.

It is pure next-token prediction, so give it text to continue rather than chat messages. Start here if you want to run your own SFT or preference tuning, or to study what pretraining alone produces.

Here is what Apex-2-Base offers:

Token efficiency. With 1/23 to 1/400 of the pretraining data of comparable base models, its code score matches Qwen2.5-1.5B, trained on 18T tokens: HumanEval+ 32.9. The largest gaps are math and commonsense, which track pretraining tokens.

Pretraining. Three stages, 54,250 steps in all. 20.0B tokens on one GH200; a 46.1B-token continuation that moved to two GH200s through DiLoCo; then 20.4B tokens of linear learning-rate decay. Global batch of 1M tokens, 8-bit AdamW, BF16 compute with FP32 master weights.

Data. Forty deduplicated sources: web, code in 11 languages, math, papers, StackExchange and synthetic text. The decay stage shifted the mix toward code and math (web 30, code 35, math 15, curated 20).

What fine-tuning adds. Two SFT stages turned this model into Apex-2, lifting HumanEval from 36.6 to 43.9 while commonsense stayed within two points.

Base models compared with base models, right after pretraining. On code, Apex-2-Base keeps pace with models trained on far more data; on commonsense it trails.

	Apex-2-Base3.87B · 1.45B active	Qwen2.5-1.5B1.5B	Qwen2.5-Coder-1.5B1.5B	Gemma-2-2B2B	OLMoE-1B-7B7B · 1.3B active
Pretraining dataTokens	0.087T	18T	5.5T	2T	5.1T
CodeHumanEval	36.6	37.2	43.9	17.7	—
CodeHumanEval+	32.9	32.9	36.6	—	—
CodeMBPP	54.8	60.2	69.2	29.6	—
CodeMBPP+	46.3	49.6	58.6	—	—
CommonsenseHellaSwag	60.8	67.9	61.8	73.0	81.7
CommonsensePIQA	73.9	—	—	77.8	78.7
CommonsenseWinoGrande	60.1	65.0	60.7	70.9	70.6

Apex-2-Base was measured by us (EvalPlus for code; lm-eval loglikelihood without a chat template for commonsense). Other numbers come from official model cards and reports, and protocols differ between models, so treat this as a rough comparison. Sources: Qwen2.5 blog and report, Qwen2.5-Coder report, Gemma 2 2B, OLMoE base.

All results, with Apex-2 after SFT

	Apex-2-Basepretrained	Apex-2after SFT
CodeHumanEval	36.6	43.9
CodeHumanEval+	32.9	41.5
CodeMBPP	54.8	56.3
CodeMBPP+	46.3	48.9
MathGSM8K	15.18-shot	32.40-shot CoT
CommonsenseHellaSwag	60.8	62.2
CommonsenseARC-e	66.7	64.1
CommonsenseARC-c	39.7	38.4
CommonsensePIQA	73.9	74.8
CommonsenseWinoGrande	60.1	61.8
CommonsenseLAMBADA	54.5	52.8

The two GSM8K scores use different protocols (8-shot for the base model, 0-shot chain of thought after SFT), so they are not directly comparable.

Quick start

generate.py

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOON1v/Apex-2-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto")

# A base model continues text: give it a prefix, not a chat message
ids = tok("def is_prime(n):", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128)
print(tok.decode(out[0], skip_special_tokens=True))

Before you download

Apex-2-Base or Apex-2?

Use Apex-2 for chat, code and instruction following out of the box. Use Apex-2-Base to run your own post-training, or to study the model before fine-tuning.

What are its limits?

It does not follow instructions; it continues text. It is English-centric, its knowledge is limited, and its context is 4,096 tokens.

Is it free to use?

The weights are free to download from Hugging Face. The license and intended use are on the model card; please read it before deploying.

Introducing Apex-2