What are diffusion models? How AI image generation works
Diffusion models generate images by removing noise step by step. How the forward and reverse processes work, with a full DDPM sampler in plain Python.

A diffusion model is a generative model that learns to turn random noise into data by removing a little noise at a time. During training you take real images, add Gaussian noise in many small steps until nothing is left, and train a neural network to predict the noise that was added; to generate, you start from pure noise and run that network backwards, step by step, until an image appears. Stable Diffusion, which its authors describe as “a latent text-to-image diffusion model”, is the best-known open example.
Where did diffusion models come from?
The idea is older than the hype. Sohl-Dickstein et al. (2015) borrowed it from nonequilibrium thermodynamics: “systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process”, then learn the reverse process that restores it. It worked, but the samples were not yet competitive with the best GANs.
Two papers in 2020 made it practical. Ho, Jain and Abbeel’s DDPM (June 2020) simplified the training objective to predicting the added noise with a U-Net, and reported an FID of 3.17 on CIFAR-10, state of the art at the time. Song et al. (November 2020) recast the same process as a stochastic differential equation and showed the reverse process needs only the score, the gradient of the log density of the noisy data, which a network can learn.
Then the engineering papers turned it into products:
- Latent diffusion (Rombach et al., December 2021) runs the diffusion process in “the latent space of powerful pretrained autoencoders” instead of on raw pixels, which cuts compute enough to run on consumer GPUs. It also adds cross-attention layers so text can steer generation. Stable Diffusion builds on it: its first version paired a downsampling-factor-8 autoencoder with an 860M-parameter U-Net and a CLIP ViT-L/14 text encoder.
- Classifier-free guidance (Ho and Salimans, July 2022) trains one network both with and without the text condition and blends the two predictions at sampling time. The guidance scale knob in image tools is this blend.
- Diffusion transformers (Peebles and Xie, December 2022) replace the U-Net with “a transformer that operates on latent patches” and show quality improving as the transformer grows. Their DiT-XL/2 reached an FID of 2.27 on ImageNet 256 by 256.
How do diffusion models work?
There are two processes, and only one of them is learned.
The forward process is fixed. At each step t you scale the current sample down slightly and add a little Gaussian noise. DDPM uses T = 1000 steps with the noise variance beta rising linearly from 0.0001 to 0.02. Because every step is Gaussian, you can jump straight to any step in closed form: x_t = sqrt(abar_t) * x_0 + sqrt(1 - abar_t) * noise, where abar_t is the running product of 1 - beta. By step 1000 almost nothing of the original is left.
The reverse process is learned. A network eps(x_t, t) looks at a noisy sample and the step number and predicts the noise in it. DDPM’s training loop is short: pick a real image, pick a random t, add noise to get x_t, and minimize the squared error between the true noise and the predicted one. To sample, start from pure noise and at each step subtract a scaled version of the predicted noise, then add back a small amount of fresh noise, a thousand times.
Text conditioning goes into the network as an extra input, usually through cross-attention to a text encoder’s embeddings, so the prediction becomes “the noise in this image, given that it should look like a red bicycle on a beach.”
Can I run a diffusion model in plain Python?
Yes, if the data is simple enough that you do not need a neural network. The script below implements the full DDPM sampler with the paper’s schedule (T = 1000, linear betas from 0.0001 to 0.02) on one-dimensional data: a mix of two bumps at -2 and +2. For data this simple the ideal noise prediction can be computed exactly, so predict_noise stands in for the trained network. Everything else is the real algorithm.
# diffusion_1d.py: a complete DDPM sampler in one dimension. Python 3, stdlib only.
# Data: a mix of two bumps at -2 and +2. In a real model a neural network predicts
# the noise; here the data is simple enough to compute the ideal prediction exactly.
import math, random
random.seed(0)
T = 1000 # steps, as in Ho et al. (2020)
betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)] # linear schedule
alphas = [1 - b for b in betas]
abar, prod = [], 1.0
for a in alphas:
prod *= a
abar.append(prod)
MUS, S = (-2.0, 2.0), 0.3 # the "dataset"
def predict_noise(x, t):
"""Ideal noise prediction E[eps | x_t] for the two-bump data."""
ab = abar[t]
var = ab * S**2 + (1 - ab) # variance of x_t per bump
w = [math.exp(-(x - math.sqrt(ab) * m) ** 2 / (2 * var)) for m in MUS]
x0 = sum(wi * (m + math.sqrt(ab) * S**2 / var * (x - math.sqrt(ab) * m))
for wi, m in zip(w, MUS)) / sum(w) # E[x_0 | x_t]
return (x - math.sqrt(ab) * x0) / math.sqrt(1 - ab)
print("forward process: how much of the original signal is left")
for t in (0, 249, 499, 749, 999):
print(f" t={t + 1:>4} sqrt(abar)={math.sqrt(abar[t]):.3f} noise std={math.sqrt(1 - abar[t]):.3f}")
def histogram(xs, label):
print(label)
for lo in range(-4, 4):
n = sum(lo <= x < lo + 1 for x in xs)
print(f" [{lo:+d},{lo + 1:+d}) {'#' * (n // 20)}")
xs = [random.gauss(0, 1) for _ in range(1000)] # start from pure noise
histogram(xs, "step 1000: pure Gaussian noise")
for t in reversed(range(T)): # reverse process
for i, x in enumerate(xs):
eps = predict_noise(x, t)
mean = (x - betas[t] / math.sqrt(1 - abar[t]) * eps) / math.sqrt(alphas[t])
xs[i] = mean + (math.sqrt(betas[t]) * random.gauss(0, 1) if t > 0 else 0)
histogram(xs, "step 0: generated samples")
left = [x for x in xs if x < 0]
print(f"left bump {len(left)}, right bump {len(xs) - len(left)}, "
f"left mean {sum(left) / len(left):+.2f}")
Output (Python 3.13.11):
forward process: how much of the original signal is left
t= 1 sqrt(abar)=1.000 noise std=0.010
t= 250 sqrt(abar)=0.724 noise std=0.690
t= 500 sqrt(abar)=0.280 noise std=0.960
t= 750 sqrt(abar)=0.058 noise std=0.998
t=1000 sqrt(abar)=0.006 noise std=1.000
step 1000: pure Gaussian noise
[-4,-3)
[-3,-2)
[-2,-1) ######
[-1,+0) ##################
[+0,+1) ################
[+1,+2) ######
[+2,+3) #
[+3,+4)
step 0: generated samples
[-4,-3)
[-3,-2) ############
[-2,-1) ############
[-1,+0)
[+0,+1)
[+1,+2) ############
[+2,+3) ############
[+3,+4)
left bump 502, right bump 498, left mean -2.00
The first block shows the schedule doing its job: at step 500 only 28% of the original signal is left, and by step 1000 it is 0.6%, so the endpoint is indistinguishable from pure noise. The second block starts from a single Gaussian hump centered on zero. A thousand small denoising steps later, the same 1,000 points have split into two bumps centered exactly on -2 and +2, with an even split between them. No sample was copied from the data; each one was pushed there by the noise predictions.
A real image model does the same thing in a space with tens of thousands of dimensions or more (an image, or its compressed latent), where the ideal prediction cannot be written down, so a network with billions of parameters learns an approximation of it.
Why does sampling take so many steps?
Each reverse step only removes a little noise, because the Gaussian approximation behind it only holds for small steps. The script above calls predict_noise 1,000 times per sample. With a neural network, each call is a full forward pass, which is why early diffusion models were slow next to a GAN that generates in one pass. Much of the research since 2020 is about cutting that count: deterministic samplers that take larger steps, distillation into few-step students, and running in a compressed latent space.
Diffusion vs autoregressive models: what’s the difference?
An autoregressive model, like every chat LLM, writes one token at a time, left to right, and never revisits a token once it is written (the loop I walked through in what are tokens). A diffusion model produces the whole output at once and refines all of it in every step.
That difference is why text diffusion has become a research direction. LLaDA (February 2025) trained an 8B text diffusion model from scratch and reports it “competitive with strong LLMs like LLaMA3 8B in in-context learning.” Google DeepMind’s Gemini Diffusion is an experimental text diffusion model that generates blocks of tokens at once, with a reported average sampling speed of 1,479 tokens per second. For images and video, diffusion is the default; for text, autoregressive transformers still are, and the open question is whether parallel refinement can match them on reasoning.
Why do diffusion models matter to a builder?
Most builders will call a hosted image or video model, not train one. Knowing the mechanism still pays off in three places. The guidance scale and step count are the two knobs you will be offered, and they map directly onto classifier-free guidance and the reverse loop. Cost and latency scale with the number of steps, unlike a chat model where they scale with output tokens. And image editing, inpainting and “image to image” are all the same trick: start the reverse process from a partially noised version of your image instead of from pure noise.
What I would do
For product work, I would use a hosted image model and spend my effort on the prompt, the reference images and an approval step, because generated images carry brand and legal risk that text rarely does. I would log the model version, seed, steps and guidance scale for every generated asset so it can be reproduced or withdrawn. For learning, I would run the script above and then change S or MUS to see the sampler find any shape you give it. If I needed fine control over a house style, I would look at fine-tuning an open latent diffusion model with LoRA before building anything bigger.
FAQ
How do diffusion models generate images from text?
A text encoder turns the prompt into embeddings. The denoising network reads them through cross-attention at every step, so each noise prediction is conditioned on the prompt. Classifier-free guidance then pushes the result further toward the prompt by blending the conditioned and unconditioned predictions.
What is the difference between diffusion models and GANs?
A GAN trains a generator against a discriminator and produces an image in one forward pass. A diffusion model trains with a simple regression loss on the noise and generates over many steps. Diffusion training is simpler to get right, and in 2021 Dhariwal and Nichol showed diffusion beating the best GANs on ImageNet sample quality (FID 4.59 at 256 by 256).
What is latent diffusion?
Diffusion run in the compressed latent space of a pretrained autoencoder instead of on pixels. It is far cheaper, and the decoder turns the final latent back into an image. Stable Diffusion is a latent diffusion model.
Can diffusion models generate text?
Yes, as research and experimental products: LLaDA and Gemini Diffusion generate text by refining many tokens in parallel. Mainstream chat models are still autoregressive.
Sources
- Sohl-Dickstein et al., Deep Unsupervised Learning using Nonequilibrium Thermodynamics, March 12, 2015
- Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models, June 19, 2020 (T = 1000, linear betas from 0.0001 to 0.02, section 4)
- Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, November 26, 2020
- Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, December 20, 2021
- Ho and Salimans, Classifier-Free Diffusion Guidance, July 26, 2022
- Dhariwal and Nichol, Diffusion Models Beat GANs on Image Synthesis, May 11, 2021
- CompVis, stable-diffusion README (read 2026-10-31)
- Peebles and Xie, Scalable Diffusion Models with Transformers, December 19, 2022
- Nie et al., Large Language Diffusion Models (LLaDA), February 14, 2025
- Google DeepMind, Gemini Diffusion (read 2026-10-31)