Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

How are LLMs trained? Pretraining to post-training, explained

How LLMs are trained: pretraining on trillions of tokens, then supervised fine-tuning, RLHF or DPO, and RL with verifiable rewards. With a compute calculator.

How are LLMs trained? Pretraining to post-training, explained

LLMs are trained in two phases. Pretraining teaches a transformer to predict the next token over trillions of tokens of text and code, which produces a base model that knows a lot but only continues text. Post-training then turns that base model into an assistant with supervised fine-tuning on example conversations, preference training (RLHF or DPO) and, since 2025, reinforcement learning on tasks with checkable answers.

What happens during pretraining?

Pretraining is self-supervised: the text is its own label. Take a document, hide the next token, ask the model to predict it, measure how much probability it gave the right answer (the cross-entropy loss), and nudge every weight to make that probability higher. Repeat for every position of every document.

The pieces, in order:

  1. Collect and clean data. Web pages, code, books, papers and multilingual text, deduplicated and filtered for quality, personal data and unsafe content.
  2. Build a tokenizer and convert the corpus into token IDs. Llama 3 uses a 128K-token vocabulary; tokens are the unit everything else is counted in.
  3. Train the network. A transformer with randomly initialized weights reads batches of sequences, predicts every next token in parallel (the causal mask stops it from peeking), and updates its weights by gradient descent.

Meta’s Llama 3 paper is the most detailed public account of doing this at frontier scale. The flagship model has 405 billion parameters and was pretrained on 15.6 trillion text tokens using 3.8 × 10^25 FLOPs, on up to 16,000 H100 GPUs. The final data mix is roughly 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual tokens. Meta chose the mix by training small models on candidate mixes and extrapolating with scaling laws.

Two details from that paper are worth knowing because they’re the parts that don’t make it into launch posts. First, the end of pretraining is deliberate. There’s a long-context stage (about 800B tokens for Llama 3) that stretches the context window, then annealing: on the final 40M tokens the learning rate is brought linearly to zero while the data mix upsamples high-quality sources, Meta reports that annealing on high-quality code and math data lifted the 8B model’s GSM8k and MATH validation scores by 24.0% and 6.4%, with negligible gains at 405B. Second, failures are routine. During a 54-day snapshot of the 405B run there were 466 job interruptions, 419 of them unexpected, and about 78% of the unexpected ones were attributed to confirmed or suspected hardware issues.

How much compute does pretraining take?

A standard estimate from Kaplan et al.’s scaling laws paper is about 6 FLOPs per parameter per training token: 2 for the forward pass and roughly twice that for the backward pass. So compute is about 6 * N * D. This script checks that rule against three published runs.

# training_compute.py: check published training budgets with C = 6 * N * D
# (about 6 FLOPs per parameter per training token). Python 3, stdlib only.
runs = [  # name, parameters N, tokens D, FLOPs the paper reports
    ("GPT-3 175B (2020)",      175e9,   300e9, 3.14e23),
    ("Chinchilla 70B (2022)",   70e9,   1.4e12, None),
    ("Llama 3 405B (2024)",    405e9, 15.6e12, 3.8e25),
]
for name, n, d, reported in runs:
    c = 6 * n * d
    note = f"paper: {reported:.2e}" if reported else "paper: same budget as Gopher"
    print(f"{name:<23} 6ND = {c:.2e} FLOPs   {d / n:>4.0f} tokens per parameter   {note}")

# What does 3.8e25 FLOPs mean in GPU time? The Llama 3 paper (Table 4) reports
# 380 to 430 TFLOP/s of useful work per H100; take 400 on 16,384 GPUs.
gpu_seconds = 3.8e25 / 400e12
print(f"\nLlama 3 405B at 400 TFLOP/s per GPU: {gpu_seconds / 3600 / 1e6:.1f}M GPU-hours,"
      f" or {gpu_seconds / 86400 / 16_384:.0f} days on 16,384 GPUs")

Output (Python 3.13.11):

GPT-3 175B (2020)       6ND = 3.15e+23 FLOPs      2 tokens per parameter   paper: 3.14e+23
Chinchilla 70B (2022)   6ND = 5.88e+23 FLOPs     20 tokens per parameter   paper: same budget as Gopher
Llama 3 405B (2024)     6ND = 3.79e+25 FLOPs     39 tokens per parameter   paper: 3.80e+25

Llama 3 405B at 400 TFLOP/s per GPU: 26.4M GPU-hours, or 67 days on 16,384 GPUs

The rule matches both published figures to within 1%. The middle column is the more interesting one: GPT-3 saw about 2 tokens per parameter, Chinchilla 20, Llama 3 405B 39. The field learned, through the Chinchilla paper in 2022, that models had been undertrained, and data per parameter has gone up ever since. Smaller models go much further: Meta’s Llama 3 announcement says its 8B model kept improving log-linearly up to 15T tokens, against a Chinchilla-optimal amount of about 200B.

The last line is my arithmetic, using the per-GPU throughput the paper reports in its Table 4 (430, 400 and 380 TFLOPs per GPU across its three configurations): roughly 26 million GPU-hours of useful work, or about two months on 16,384 GPUs, before counting restarts and the failures above.

What is post-training?

A base model completes text. Ask it a question and it may answer, or continue with more questions, because both happen on the web. Post-training teaches it to behave like an assistant: follow instructions, use a chat format, refuse some requests, call tools and stop when done.

OpenAI’s InstructGPT paper (March 2022) set the template, in three steps:

  1. Supervised fine-tuning (SFT). Labelers wrote demonstrations of good responses, and GPT-3 was fine-tuned on them. The SFT set had about 13k training prompts.
  2. Reward model. Labelers ranked several model outputs for the same prompt, and a separate model was trained to predict those rankings (33k training prompts).
  3. Reinforcement learning. The SFT model was optimized against the reward model with PPO (31k prompts). This is RLHF, reinforcement learning from human feedback, which has its own post: what RLHF is.

OpenAI hired a team of 40 contractors to label the data. The payoff was large: human raters preferred outputs from the 1.3B InstructGPT model to those of the 175B GPT-3, “despite having 100x fewer parameters”. Post-training doesn’t add much knowledge; it changes what the model does with the knowledge it has.

The reward model and the RL loop are expensive and finicky, and the DPO paper (May 2023) offered a shortcut. Direct Preference Optimization trains on the same preference pairs with a simple classification loss, no separate reward model and no sampling during training, and the authors report it matching or beating PPO-based RLHF on their tasks. Llama 3’s post-training uses SFT, rejection sampling and DPO, applied in six rounds, each collecting fresh preference data on the latest model. The trade-offs are in DPO vs RLHF.

What is RL with verifiable rewards?

Preference training needs humans, or a model imitating humans, to judge which answer is better. For math and code there’s a cheaper judge: check the answer. Did the final number match? Do the tests pass?

DeepSeek’s R1 paper (January 2025) showed how far that goes. DeepSeek-R1-Zero was trained with large-scale reinforcement learning on the base model “without supervised fine-tuning (SFT) as a preliminary step”, using rule-based rewards: an accuracy reward for a correct, verifiable answer and a format reward for putting its reasoning between think tags. Its pass@1 score on AIME 2024 rose from 15.6% to 71.0% during training. The released R1 added a small amount of cold-start SFT data and more stages on top. Nobody told the model to think longer, yet the paper’s Figure 3 shows its average response length growing steadily through RL: it learned to spend more tokens on a problem because that earned reward. That’s the mechanism behind reasoning models. I wrote about the release itself in DeepSeek R1, and the technique gets its own post in what RLVR is.

So a 2026 training pipeline usually looks like this: pretrain, anneal on high-quality data, extend the context, SFT, preference optimization, then RL on verifiable tasks, often with several rounds of each.

What I would do

Almost nobody reading this will pretrain a model, and that’s fine. What the pipeline tells a builder is where behavior comes from.

Knowledge comes from pretraining. If a model doesn’t know your internal APIs or last month’s pricing, no amount of prompt wording will fix it; put the facts in the context with RAG or tools.

Behavior comes from post-training. Format, tone, refusal style, tool-calling habits and how much the model reasons are all trained in, which is why two models with similar benchmark scores feel different in an agent loop. If you need to change behavior at scale, fine-tuning is the lever, and SFT on a few thousand clean examples is the first thing I’d try.

From a data-platform seat, the line I’d circle is the Llama 3 data section. Deduplication, quality classifiers, PII filtering, a data mix chosen by experiment: that’s a data pipeline with a GPU cluster attached. The model is downstream of the data, and so are its problems.

FAQ

How long does it take to train an LLM?

For frontier models, months. The Llama 3 paper describes a 54-day snapshot of a much longer 405B pretraining run on up to 16,000 GPUs, plus post-training afterwards. A small model fine-tuned on your data can take hours on one GPU.

What data are LLMs trained on?

Mostly public web text and code, filtered and deduplicated. Llama 3’s announcement says its pretraining data was “collected from publicly available sources”, and the paper describes filters for personal data and unsafe sites. Post-training adds human-written demonstrations, preference rankings and, increasingly, model-generated data that’s been checked.

What is the difference between pretraining and fine-tuning?

Pretraining starts from random weights and learns general language from trillions of tokens. Fine-tuning starts from a trained model and adjusts it on a much smaller, targeted dataset to change its behavior or specialize it.

Do LLMs keep learning from my conversations?

Not during the conversation; the weights are fixed at inference time. Whether your conversations are used to train future models depends on the provider’s data policy and your settings, so read it for any API or app you use.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.