Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What are scaling laws? Chinchilla and LLM compute, explained

Scaling laws predict how an LLM's loss falls as you add parameters, data and compute. What Kaplan and Chinchilla found, and why models now overtrain.

What are scaling laws? Chinchilla and LLM compute, explained

Scaling laws are empirical formulas that predict a language model’s loss from three inputs: the number of parameters, the number of training tokens and the training compute. Loss falls as a smooth power law in each, which lets a lab train small models, fit the curve and predict how a model 100 times bigger will do before spending the money. The best-known result is DeepMind’s Chinchilla paper (2022): for a fixed compute budget, parameters and training tokens should grow in equal proportion, roughly 20 tokens per parameter.

What did the first scaling laws paper find?

OpenAI’s Scaling Laws for Neural Language Models (Kaplan et al., January 2020) trained transformers across a wide range of sizes and found that “the loss scales as a power-law with model size, dataset size, and the amount of compute used for training”, with some trends spanning more than seven orders of magnitude.

The fitted forms look like this, with N counting non-embedding parameters and D counting tokens:

L(N) = (N_c / N) ^ 0.076      when data is not the limit
L(D) = (D_c / D) ^ 0.095      when the model is not the limit

Small exponents mean slow but very steady improvement: halving the loss takes an enormous multiple of scale, yet the curve doesn’t bend away. The paper also found that architecture details such as width versus depth mattered little across a wide range, that larger models are more sample-efficient, and that compute-efficient training means training very large models and “stopping significantly before convergence”.

That last finding shaped the next two years. Kaplan’s fit said that as compute grows, most of it should go to a bigger model (optimal size growing as compute to the 0.73) and relatively little to more data. GPT-3, published four months later at 175 billion parameters and 300 billion tokens, fits that advice: about 2 tokens per parameter.

What is the Chinchilla scaling law?

DeepMind’s Training Compute-Optimal Large Language Models (Hoffmann et al., March 2022) redid the experiment more carefully: over 400 models from 70 million to over 16 billion parameters, trained on 5 to 500 billion tokens. Three different estimation methods all pointed the same way: “the model size and the number of training tokens should be scaled equally”. Double the compute, and you should grow both parameters and tokens by about 1.4 times.

The paper’s Table 3 turns that into concrete targets: about 20 billion tokens for a 1B model, 205 billion for 10B, 1.5 trillion for 67B. By that yardstick the large models of 2021 were badly undertrained: most had been trained on roughly 300 billion tokens regardless of size.

To test it, DeepMind trained Chinchilla: 70 billion parameters on 1.4 trillion tokens, the same compute as their 280 billion parameter Gopher. Chinchilla “uniformly and significantly outperforms” Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG, and reached 67.5% average accuracy on MMLU, which the paper describes as more than a 7% improvement over Gopher. A model a quarter the size beat its predecessor on the same budget, and it was cheaper to serve.

How do you find the compute-optimal model size?

The Chinchilla paper’s third method fits one formula for loss as a function of both N and D:

L(N, D) = E + A / N^0.34 + B / D^0.28       E = 1.69, A = 406.4, B = 410.7

E is the irreducible loss of natural text. The other two terms are the penalties for a model that’s too small and for data that’s too little. Training compute is about 6 FLOPs per parameter per token, so for a fixed budget C you can only pick N, and D follows as C / 6N. This script sweeps N at Gopher’s budget and finds the minimum.

# chinchilla.py: find the compute-optimal model size with the Chinchilla loss fit.
# L(N, D) = E + A / N^0.34 + B / D^0.28   (Hoffmann et al. 2022, equation 10)
# Python 3, stdlib only.
E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28

def loss(n, d):
    return E + A / n**alpha + B / d**beta

C = 5.76e23                                         # Gopher's budget (Table 3: 1 "Gopher unit")
print(f"Fixed budget C = {C:.2e} FLOPs, so tokens D = C / (6N)")
for n in (280e9, 175e9, 70e9, 40e9, 20e9, 10e9, 5e9):
    d = C / (6 * n)
    print(f"  N = {n/1e9:>4.0f}B  D = {d/1e9:>6,.0f}B tokens  ({d/n:>5.0f} tokens/param)  predicted loss {loss(n, d):.4f}")

# Search a fine grid for the minimum.
best = min((loss(n, C / (6 * n)), n) for n in (10**(9 + i / 200) for i in range(600)))
n = best[1]
print(f"Optimum under this fit: N = {n/1e9:.0f}B, D = {C/(6*n)/1e9:,.0f}B tokens, loss {best[0]:.4f}")

# Overtraining: Llama 3 8B saw about 15T tokens; the Chinchilla-style budget is about 200B.
for d in (200e9, 1e12, 15e12):
    print(f"8B model, {d/1e12:>5.1f}T tokens: predicted loss {loss(8e9, d):.4f}")

Output (Python 3.13.11):

Fixed budget C = 5.76e+23 FLOPs, so tokens D = C / (6N)
  N =  280B  D =    343B tokens  (    1 tokens/param)  predicted loss 1.9840
  N =  175B  D =    549B tokens  (    3 tokens/param)  predicted loss 1.9632
  N =   70B  D =  1,371B tokens  (   20 tokens/param)  predicted loss 1.9376
  N =   40B  D =  2,400B tokens  (   60 tokens/param)  predicted loss 1.9313
  N =   20B  D =  4,800B tokens  (  240 tokens/param)  predicted loss 1.9334
  N =   10B  D =  9,600B tokens  (  960 tokens/param)  predicted loss 1.9470
  N =    5B  D = 19,200B tokens  ( 3840 tokens/param)  predicted loss 1.9732
Optimum under this fit: N = 32B, D = 2,967B tokens, loss 1.9307
8B model,   0.2T tokens: predicted loss 2.1459
8B model,   1.0T tokens: predicted loss 2.0438
8B model,  15.0T tokens: predicted loss 1.9485

Three things stand out. The curve is U-shaped: too big a model with too little data (280B) and too small a model with too much data (5B) both lose. The bottom is flat: anything from about 20B to 70B lands within 0.007 of the optimum, which is why “roughly 20 tokens per parameter” is a fine rule of thumb. And this particular fit puts the optimum at 32B, smaller than the 40 to 70B the paper’s other two methods suggested. The paper says so itself: the parametric approach “predicts even smaller models being optimal at larger compute budgets”. A 2024 replication attempt by Besiroglu et al. argued that this third fit is inconsistent with the first two methods, and their own refit agreed with the other two. Treat the constants as a sketch of the shape, not a precise prediction, and remember they describe loss on DeepMind’s dataset, not on yours.

Why do modern models train far past Chinchilla-optimal?

Look at the last three lines of the output. An 8B model trained on 15 trillion tokens is predicted to reach a loss close to the 70B model’s at Gopher’s budget, with about a ninth of the parameters. It costs more to train (6 × 8B × 15T is 7.2 × 10^23 FLOPs, about 25% over Gopher’s budget), but every token it serves afterwards is far cheaper. Chinchilla optimizes training compute. It says nothing about the cost of running the model afterwards, and inference is paid on every token, forever.

Meta made exactly this trade. Its Llama 3 announcement (April 2024) says the Chinchilla-optimal amount of training for an 8B model “corresponds to ~200B tokens”, yet both the 8B and 70B models “continued to improve log-linearly” up to 15 trillion tokens. Sardana et al.’s Beyond Chinchilla-Optimal (December 2023) put the economics into the scaling law: with large inference demand, around a billion requests, you “should train models smaller and longer than Chinchilla-optimal”.

Scaling laws are also how labs plan the flagship run. The Llama 3 paper fits its own law on small runs and extrapolates to its 3.8 × 10^25 FLOPs budget, which “suggests training a 402B parameter model on 16.55T tokens”. They trained 405B on 15.6T. The paper also notes the IsoFLOP curves flatten near the minimum at larger budgets, the same flat bottom the script shows.

Have scaling laws stopped working?

The pretraining curves haven’t broken; what changed is where the next unit of compute goes. Data is the constraint people worry about: Chinchilla already warned that the data needed was “far beyond what is currently used”. And since the first reasoning models in 2024, labs scale a second axis: compute spent at inference time, letting a model think longer on hard problems. That’s test-time compute, and it’s how reasoning models trade more output tokens for better answers on math and code. I covered the first of them in OpenAI o1-preview.

So in 2026 there are three knobs, not two: parameters, training tokens and thinking tokens, each with its own cost curve.

What I would do

Use scaling laws as a cost model, not a leaderboard predictor. Lower loss doesn’t map neatly to your task’s accuracy, and the published constants were fitted on someone else’s data.

Where they help a builder is in reading model sizes. A small model trained on a huge number of tokens is a deliberate inference-cost decision, and it’s usually the right one for high-volume work. From a data-platform seat, the line I’d circle is Sardana’s: total cost is training plus serving, and serving is the term that grows with your traffic. The same logic says to try the smallest model that passes your evals first, and to route to a bigger one only when the task needs it.

If you’re fine-tuning rather than pretraining, the useful takeaway is the flat bottom: within a range, more and better data does about as much as a bigger model, and data is the part you control. More on how that plays out across the pipeline in how LLMs are trained.

FAQ

What is the Chinchilla ratio?

About 20 training tokens per parameter for a compute-optimal model, from the Chinchilla paper’s Table 3 (20.2 billion tokens for a 1 billion parameter model). It minimizes loss for a fixed training budget, not total cost.

What is the difference between Kaplan and Chinchilla scaling laws?

Both find power laws. Kaplan et al. (2020) concluded that extra compute should go mostly into model size; Hoffmann et al. (2022) found model size and data should grow equally. Chinchilla’s view won, and models since have been trained on far more tokens per parameter.

What is compute-optimal training?

Choosing the model size and token count that give the lowest loss for a fixed amount of training compute. With compute at about 6 FLOPs per parameter per token, a bigger model means fewer tokens, and the optimum balances the two.

Do scaling laws apply to fine-tuning?

Similar power laws show up in many settings, but the published constants come from pretraining specific models on specific data. For fine-tuning, measure on your own task with a few data sizes and extrapolate cautiously.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.