Diffusion models are the method behind the overwhelming majority of AI image, video, and audio generation today. They work very differently from large language models: a language model writes left to right, one token after another, while a diffusion model starts with an entire frame of pure noise and repeatedly refines it into a clear picture.
The name comes from physical diffusion — a drop of ink spreading through water until it's uniform and its original shape is gone. What a diffusion model learns is the inverse of that process: how to pull the ink back out of uniform noise.
Grab It in One Sentence First
A diffusion model first learns how an image is destroyed into noise step by step, then runs that procedure in reverse to generate images out of noise.
The everyday analogy is sculpture — Michelangelo's line about the David already being inside the block, and his job being to remove everything else. A diffusion model's "block" is a field of random noise, the prompt tells it what to carve out, and it chips away whatever doesn't belong to that picture.
How It Trains and Generates
The forward process is simple and requires no learning: take a real image and add Gaussian noise by a fixed schedule until it's pure static. Because exactly how much was added at each step is known, this automatically produces vast quantities of "noisy image → the noise to remove" training pairs.
The reverse process is what the model actually learns: given a noisy image and which step it's on, predict the noise to subtract. The training objective is almost counterintuitively plain — the model's real job is recognizing noise, and generative ability is a byproduct.
At generation time it starts from pure noise and calls the model dozens of times, injecting the text prompt as a condition at every step. Step count is a direct speed-quality trade-off, and it's where the "fast / standard / high quality" tiers in consumer products come from.
Latent Space: Why It Became Practical
Early diffusion models denoised directly in pixel space, which is brutally slow at high resolution. Latent diffusion, introduced in 2022, changed the approach: an autoencoder first compresses the image into a much smaller latent representation, the whole denoising loop runs in that latent space, and the result is decoded back to pixels at the end. Compute dropped by one to two orders of magnitude, which is what made consumer GPUs viable and let open ecosystems like Stable Diffusion take off.
The text condition is injected via cross-attention: the prompt is encoded into vectors, and every denoising step lets the image "look at" that text, steering generation toward what was described.
From U-Net to DiT, and On to Video
Early diffusion backbones were convolutional U-Nets. The field then found that replacing the backbone with a Transformer works better and scales more cleanly with compute — the DiT, or Diffusion Transformer, design. Most current image and video generators follow that route, which is also how scaling law intuitions carried over to visual generation.
Video generation is the same mechanism plus a time dimension: consecutive frames are compressed into latent space together, and denoising has to respect consistency across both space and time. The hard part shifts from "does this frame look good" to "is this still the same person three seconds later."
How It Differs From Autoregressive Generation
Large language models are autoregressive: they emit one token after another, each step seeing only what's already written, and a wrong turn is hard to undo. Diffusion is iterative refinement: every step modifies the whole canvas, early steps fix the overall structure and later ones add detail, which makes global composition easier to control.
That also explains the difference in feel. Text generation visibly streams out sentence by sentence; image generation shows one picture sharpening from blur. Recent years have seen attempts to apply diffusion to text, and autoregressive approaches to images — ChatGPT's image capability belongs to the latter line of thinking — and the two families keep borrowing from each other, but diffusion remains the mainstream for images and video.
Where People Get It Wrong
"It retrieves and collages from a database." It doesn't. The weights don't store source images, and generation is numerical optimization starting from random noise. That doesn't mean copyright concerns are void: training-data provenance, close imitation of a specific artist's style, and the possibility of reproducing near-duplicates when training data is heavily repeated are all real and still contested.
"The same prompt should give the same image." The starting point is random noise, so unless the seed, model, step count, and sampler are all fixed, results differ. That's by design, not instability.
"More steps is always better." Returns fall off quickly past a certain point and can even over-smooth the image. Different samplers have very different useful ranges, so it's worth testing rather than maxing the slider.
When It's Worth Understanding
If you just use image tools, grasping "gradual convergence out of noise" explains a lot of daily behavior: why changing one word changes everything (the noise seed and the condition jointly determine the path), why hands and text come out wrong (fine detail is settled only in late steps, under weak constraint), and why an image-to-image "denoising strength" slider is really choosing which step to restart from.
If you're evaluating or integrating products, look at latent resolution, the cost curve across samplers and step counts, and the controllability toolkit (structural conditioning like ControlNet, reference-image consistency, regional inpainting). Those decide whether a model is usable far more than its name does.