Diffusion model

Short description: Technique for the generative modeling of a continuous probability distribution

This article discusses diffusion modeling of a continuous distribution. For the modeling of a discrete distribution, see Discrete diffusion model.

Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

In machine learning, diffusion models, also known as diffusion-based generative models or score-based generative models, are a class of latent variable generative models. A diffusion model consists of two major components: the forward diffusion process, and the reverse sampling process. The goal of diffusion models is to learn a diffusion process for a given dataset, such that the process can generate new elements that are distributed similarly as the original dataset. A diffusion model models data as generated by a diffusion process, whereby a new datum performs a random walk with drift through the space of all possible data.[1] A trained diffusion model can be sampled in many ways, with different efficiency and quality.

There are various equivalent formalisms, including Markov chains, denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations.[2] They are typically trained using variational inference.[3] The model responsible for denoising is typically called its "backbone". The backbone may be of any kind, but they are typically U-nets or transformers.

As of 2024[update], diffusion models are mainly used for computer vision tasks, including image denoising, inpainting, super-resolution, image generation, and video generation. These typically involve training a neural network to sequentially denoise images blurred with Gaussian noise.[1][4] The model is trained to reverse the process of adding noise to an image. After training to convergence, it can be used for image generation by starting with an image composed of random noise, and applying the network iteratively to denoise the image.

Diffusion-based image generators have seen widespread commercial interest, such as Stable Diffusion and DALL-E. These models typically combine diffusion models with other models, such as text-encoders and cross-attention modules to allow text-conditioned generation.[5]

Other than computer vision, diffusion models have also found applications in natural language processing[6] such as text generation[7] and summarization,[8] sound generation,[9] and reinforcement learning.[10][11]

## Contents

* 1 Denoising diffusion model
* 1.1 Non-equilibrium thermodynamics
* 1.2 Denoising Diffusion Probabilistic Model (DDPM)
* 1.2.1 Forward diffusion
* 1.2.2 Backward diffusion
* 1.2.3 Variational inference
* 1.2.4 Noise prediction network
* 1.3 Backward diffusion process
* 2 Score-based generative model
* 2.1 Score matching
* 2.1.1 The idea of score functions
* 2.1.2 Learning the score function
* 2.1.3 Annealing the score function
* 2.2 Continuous diffusion processes
* 2.2.1 Forward diffusion process
* 2.2.2 Backward diffusion process
* 2.3 Noise conditional score network (NCSN)
* 3 Their equivalence
* 4 Main variants
* 4.1 Noise schedule
* 4.2 Denoising Diffusion Implicit Model (DDIM)
* 4.3 Latent diffusion model (LDM)
* 4.4 Architectural improvements
* 4.5 Classifier guidance
* 4.5.1 With temperature
* 4.6 Classifier-free guidance (CFG)
* 4.7 Samplers
* 4.8 Other examples
* 5 Flow-based diffusion model
* 5.1 Optimal transport flow
* 5.2 Rectified flow
* 6 Choice of architecture
* 6.1 Diffusion model
* 6.2 Conditioning
* 6.3 Upscaling
* 7 Examples
* 7.1 OpenAI
* 7.2 Lightricks
* 7.3 Stability AI
* 7.4 Google
* 7.5 Meta
* 8 See also
* 9 Further reading
* 10 References

## Denoising diffusion model

### Non-equilibrium thermodynamics

Diffusion models were introduced in 2015 as a method to train a model that can sample from a highly complex probability distribution. They used techniques from non-equilibrium thermodynamics, especially diffusion.[12]

Consider, for example, how one might model the distribution of all naturally occurring photos. Each image is a point in the space of all images, and the distribution of naturally occurring photos is a "cloud" in space, which, by repeatedly adding noise to the images, diffuses out to the rest of the image space, until the cloud becomes all but indistinguishable from a Gaussian distribution 𝒩(0,I). A model that can approximately undo the diffusion can then be used to sample from the original distribution. This is studied in "non-equilibrium" thermodynamics, as the starting distribution is not in equilibrium, unlike the final distribution.

The equilibrium distribution is the Gaussian distribution 𝒩(0,I), with pdf ρ(x)∝e−12‖x‖2. This is just the Maxwell–Boltzmann distribution of particles in a potential well V(x)=12‖x‖2 at temperature 1. The initial distribution, being very much out of equilibrium, would diffuse towards the equilibrium distribution, making biased random steps that are a sum of pure randomness (like a Brownian walker) and gradient descent down the potential well. The randomness is necessary: if the particles were to undergo only gradient descent, then they will all fall to the origin, collapsing the distribution.

### Denoising Diffusion Probabilistic Model (DDPM)

The 2020 paper proposed the Denoising Diffusion Probabilistic Model (DDPM), which improves upon the previous method by variational inference.[3][13]

#### Forward diffusion

To present the model, some notation is required.

* β1,...,βT∈(0,1) are fixed constants.
* αt:=1−βt
* α¯t:=α1⋯αt
* σt:=1−α¯t
* σ~t:=σt−1σtβt
* μ~t(xt,x0):=αt(1−α¯t−1)xt+α¯t−1(1−αt)x0σt2
* 𝒩(μ,Σ) is the normal distribution with mean μ and variance Σ, and 𝒩(x|μ,Σ) is the probability density at x.
* A vertical bar denotes conditioning.

A forward diffusion process starts at some starting point x0∼q, where q is the probability distribution to be learned, then repeatedly adds noise to it byxt=1−βtxt−1+βtztwhere z1,...,zT are IID (Independent and identically distributed random variables) samples from 𝒩(0,I). The coefficients 1−βt and βt ensure that Var(Xt)=I assuming that Var(X0)=I. The values of βt are chosen such that for any starting distribution of x0, if it has finite second moment, then limt->∞xt|x0 converges[clarification needed] to 𝒩(0,I).

The entire diffusion process then satisfiesq(x0:T)=q(x0)q(x1|x0)⋯q(xT|xT−1)=q(x0)𝒩(x1|α1x0,β1I)⋯𝒩(xT|αTxT−1,βTI)orln⁡q(x0:T)=ln⁡q(x0)−∑t=1T12βt‖xt−1−βtxt−1‖2+Cwhere C is a normalization constant and often omitted. In particular, we note that x1:T|x0 is a Gaussian process, which affords us considerable freedom in reparameterization. For example, by standard manipulation with Gaussian process, xt|x0∼N(α¯tx0,σt2I)xt−1|xt,x0∼𝒩(μ~t(xt,x0),σ~t2I)In particular, notice that for large t, the variable xt|x0∼N(α¯tx0,σt2I) converges to 𝒩(0,I). That is, after a long enough diffusion process, we end up with some xT that is very close to 𝒩(0,I), with all traces of the original x0∼q gone.

For example, sincext|x0∼N(α¯tx0,σt2I)we can sample xt|x0 directly "in one step", instead of going through all the intermediate steps x1,x2,...,xt−1.

Derivation by reparameterization

We know xt−1|x0 is a Gaussian, and xt|xt−1 is another Gaussian. We also know that these are independent. Thus we can perform a reparameterization: xt−1=α¯t−1x0+1−α¯t−1z xt=αtxt−1+1−αtz′ where z,z′ are IID Gaussians.

There are 5 variables x0,xt−1,xt,z,z′ and two linear equations. The two sources of randomness are z,z′, which can be reparameterized by rotation, since the IID Gaussian distribution is rotationally symmetric.

By plugging in the equations, we can solve for the first reparameterization: xt=α¯tx0+αt−α¯tz+1−αtz′⏟=σtz″ where z″ is a Gaussian with mean zero and variance one.

To find the second one, we complete the rotational matrix: [z″z‴]=[αt−α¯tσtβtσt??][zz′]

Since rotational matrices are all of the form [cos⁡θsin⁡θ−sin⁡θcos⁡θ], we know the matrix must be [z″z‴]=[αt−α¯tσtβtσt−βtσtαt−α¯tσt][zz′] and since the inverse of rotational matrix is its transpose,
[zz′]=[αt−α¯tσt−βtσtβtσtαt−α¯tσt][z″z‴]

Plugging back, and simplifying, we have xt=α¯tx0+σtz″ xt−1=μ~t(xt,x0)−σ~tz‴

#### Backward diffusion

The key idea of DDPM is to use a neural network parametrized by θ. The network takes in two arguments xt,t, and outputs a vector μθ(xt,t) and a matrix Σθ(xt,t), such that each step in the forward diffusion process can be approximately undone by xt−1∼𝒩(μθ(xt,t),Σθ(xt,t)). This then gives us a backward diffusion process pθ defined bypθ(xT)=𝒩(xT|0,I)pθ(xt−1|xt)=𝒩(xt−1|μθ(xt,t),Σθ(xt,t))The goal now is to learn the parameters θ such that pθ(x0) is as close to q(x0) as possible. To do that, we use maximum likelihood estimation with variational inference.

#### Variational inference

The ELBO inequality states that ln⁡pθ(x0)≥Ex1:T∼q(⋅|x0)[ln⁡pθ(x0:T)−ln⁡q(x1:T|x0)], and taking one more expectation, we getEx0∼q[ln⁡pθ(x0)]≥Ex0:T∼q[ln⁡pθ(x0:T)−ln⁡q(x1:T|x0)]We see that maximizing the quantity on the right would give us a lower bound on the likelihood of observed data. This allows us to perform variational inference.

Define the loss functionL(θ):=−Ex0:T∼q[ln⁡pθ(x0:T)−ln⁡q(x1:T|x0)]and now the goal is to minimize the loss by stochastic gradient descent. The expression may be simplified to[14]L(θ)=∑t=1TExt−1,xt∼q[−ln⁡pθ(xt−1|xt)]+Ex0∼q[DKL(q(xT|x0)‖pθ(xT))]+Cwhere C does not depend on the parameter, and thus can be ignored. Since pθ(xT)=𝒩(xT|0,I) also does not depend on the parameter, the term Ex0∼q[DKL(q(xT|x0)‖pθ(xT))] can also be ignored. This leaves just L(θ)=∑t=1TLt with Lt=Ext−1,xt∼q[−ln⁡pθ(xt−1|xt)] to be minimized.

#### Noise prediction network

Since xt−1|xt,x0∼𝒩(μ~t(xt,x0),σ~t2I), this suggests that we should use μθ(xt,t)=μ~t(xt,x0); however, the network does not have access to x0, and so it has to estimate it instead. Now, since xt|x0∼N(α¯tx0,σt2I), we may write xt=α¯tx0+σtz, where z is some unknown Gaussian noise. Now we see that estimating x0 is equivalent to estimating z.

Therefore, let the network output a noise vector ϵθ(xt,t), and let it predictμθ(xt,t)=μ~t(xt,xt−σtϵθ(xt,t)α¯t)=xt−ϵθ(xt,t)βt/σtαtIt remains to design Σθ(xt,t). The DDPM paper suggested not learning it (since it resulted in "unstable training and poorer sample quality"), but fixing it at some value Σθ(xt,t)=ζt2I, where either ζt2=βt or σ~t2 yielded similar performance.

With this, the loss simplifies to Lt=βt22αtσt2ζt2Ex0∼q;z∼𝒩(0,I)[‖ϵθ(xt,t)−z‖2]+Cwhich may be minimized by stochastic gradient descent. The paper noted empirically that an even simpler loss functionLsimple,t=Ex0∼q;z∼𝒩(0,I)[‖ϵθ(xt,t)−z‖2]resulted in better models.

### Backward diffusion process

After a noise prediction network is trained, it can be used for generating data points in the original distribution in a loop as follows:

1. Compute the noise estimate ϵ<-ϵθ(xt,t)
2. Compute the original data estimate x~0<-(xt−σtϵ)/α¯t
3. Sample the previous data xt−1∼𝒩(μ~t(xt,x~0),σ~t2I)
4. Change time t<-t−1

## Score-based generative model

Score-based generative model is another formulation of diffusion modelling. They are also called noise conditional score network (NCSN) or score-matching with Langevin dynamics (SMLD).[15][16][17][18]

### Score matching

#### The idea of score functions

Consider the problem of image generation. Let x represent an image, and let q(x) be the probability distribution over all possible images. If we have q(x) itself, then we can say for certain how likely a certain image is. However, this is intractable in general.

Most often, we are uninterested in knowing the absolute probability of a certain image. Instead, we are usually only interested in knowing how likely a certain image is compared to its immediate neighbors — e.g. how much more likely is an image of cat compared to some small variants of it? Is it more likely if the image contains two whiskers, or three, or with some Gaussian noise added?

Consequently, we are actually quite uninterested in q(x) itself, but rather, ∇xln⁡q(x). This has two major effects:

* One, we no longer need to normalize q(x), but can use any q~(x)=Cq(x), where C=∫q~(x)dx>0 is any unknown constant that is of no concern to us.
* Two, we are comparing q(x) neighbors q(x+dx), by q(x)q(x+dx)=e−⟨∇xln⁡q,dx⟩

Let the score function be s(x):=∇xln⁡q(x); then consider what we can do with s(x).

As it turns out, s(x) allows us to sample from q(x) using thermodynamics. Specifically, if we have a potential energy function U(x)=−ln⁡q(x), and a lot of particles in the potential well, then the distribution at thermodynamic equilibrium is the Boltzmann distribution qU(x)∝e−U(x)/kBT=q(x)1/kBT. At temperature kBT=1, the Boltzmann distribution is exactly q(x).

Therefore, to model q(x), we may start with a particle sampled at any convenient distribution (such as the standard Gaussian distribution), then simulate the motion of the particle forwards according to the Langevin equation dxt=−∇xtU(xt)dt+dWt and the Boltzmann distribution is, by Fokker-Planck equation, the unique thermodynamic equilibrium. So no matter what distribution x0 has, the distribution of xt converges in distribution to q as t->∞.

#### Learning the score function

Given a density q, we wish to learn a score function approximation fθ≈∇ln⁡q. This is score matching.[19] Typically, score matching is formalized as minimizing Fisher divergence function Eq[‖fθ(x)−∇ln⁡q(x)‖2]. By expanding the integral, and performing an integration by parts, Eq[‖fθ(x)−∇ln⁡q(x)‖2]=Eq[‖fθ‖2+2∇⋅fθ]+Cgiving us a loss function, also known as the Hyvärinen scoring rule, that can be minimized by stochastic gradient descent.

#### Annealing the score function

Suppose we need to model the distribution of images, and we want x0∼𝒩(0,I), a white-noise image. Now, most white-noise images do not look like real images, so q(x0)≈0 for large swaths of x0∼𝒩(0,I). This presents a problem for learning the score function, because if there are no samples around a certain point, then we can't learn the score function at that point. If we do not know the score function ∇xtln⁡q(xt) at that point, then we cannot impose the time-evolution equation on a particle:dxt=∇xtln⁡q(xt)dt+dWtTo deal with this problem, we perform annealing. If q is too different from a white-noise distribution, then progressively add noise until it is indistinguishable from one. That is, we perform a forward diffusion, then learn the score function, then use the score function to perform a backward diffusion.

### Continuous diffusion processes

#### Forward diffusion process

Consider again the forward diffusion process, but this time in continuous time:xt=1−βtxt−1+βtztBy taking the βt->β(t)dt,dtzt->dWt limit, we obtain a continuous diffusion process, in the form of a stochastic differential equation:dxt=−12β(t)xtdt+β(t)dWtwhere Wt is a Wiener process (multidimensional Brownian motion).

Now, the equation is exactly a special case of the overdamped Langevin equationdxt=−DkBT(∇xU)dt+2DdWtwhere D is diffusion tensor, T is temperature, and U is potential energy field. If we substitute in D=12β(t)I,kBT=1,U=12‖x‖2, we recover the above equation. This explains why the phrase "Langevin dynamics" is sometimes used in diffusion models.

Now the above equation is for the stochastic motion of a single particle. Suppose we have a cloud of particles distributed according to q at time t=0, then after a long time, the cloud of particles would settle into the stable distribution of 𝒩(0,I). Let ρt be the density of the cloud of particles at time t, then we haveρ0=q;ρT≈𝒩(0,I)and the goal is to somehow reverse the process, so that we can start at the end and diffuse back to the beginning.

By Fokker-Planck equation, the density of the cloud evolves according to∂tln⁡ρt=12β(t)(n+(x+∇ln⁡ρt)⋅∇ln⁡ρt+Δln⁡ρt)where n is the dimension of space, and Δ is the Laplace operator. Equivalently,∂tρt=12β(t)(∇⋅(xρt)+Δρt)

#### Backward diffusion process

If we have solved ρt for time t∈[0,T], then we can exactly reverse the evolution of the cloud. Suppose we start with another cloud of particles with density ν0=ρT, and let the particles in the cloud evolve according to

dyt=12β(T−t)ytdt+β(T−t)∇ytln⁡ρT−t(yt)⏟score function dt+β(T−t)dWt

then by plugging into the Fokker-Planck equation, we find that ∂tρT−t=∂tνt. Thus this cloud of points is the original cloud, evolving backwards.[20]

### Noise conditional score network (NCSN)

At the continuous limit, α¯t=(1−β1)⋯(1−βt)=e∑iln⁡(1−βi)->e−∫0tβ(t)dt and so xt|x0∼N(e−12∫0tβ(t)dtx0,(1−e−∫0tβ(t)dt)I) In particular, we see that we can directly sample from any point in the continuous diffusion process without going through the intermediate steps, by first sampling x0∼q,z∼𝒩(0,I), then get xt=e−12∫0tβ(t)dtx0+(1−e−∫0tβ(t)dt)z. That is, we can quickly sample xt∼ρt for any t≥0.

Now, define a certain probability distribution γ over [0,∞), then the score-matching loss function is defined as the expected Fisher divergence: L(θ)=Et∼γ,xt∼ρt[‖fθ(xt,t)‖2+2∇⋅fθ(xt,t)] After training, fθ(xt,t)≈∇ln⁡ρt, so we can perform the backwards diffusion process by first sampling xT∼𝒩(0,I), then integrating the SDE from t=T to t=0: xt−dt=xt+12β(t)xtdt+β(t)fθ(xt,t)dt+β(t)dWt This may be done by any SDE integration method, such as Euler–Maruyama method.

The name "noise conditional score network" is explained thus:

* "network", because fθ is implemented as a neural network.
* "score", because the output of the network is interpreted as approximating the score function ∇ln⁡ρt.
* "noise conditional", because ρt is equal to ρ0 blurred by an added Gaussian noise that increases with time, and so the score function depends on the amount of noise added.

## Their equivalence

DDPM and score-based generative models are equivalent.[16][1][21] This means that a network trained using DDPM can be used as a NCSN, and vice versa.

We know that xt|x0∼N(α¯tx0,σt2I), so by Tweedie's formula, we have ∇xtln⁡q(xt)=1σt2(−xt+α¯tEq[x0|xt]) As described previously, the DDPM loss function is ∑tLsimple,t with Lsimple,t=Ex0∼q;z∼𝒩(0,I)[‖ϵθ(xt,t)−z‖2] where xt=α¯tx0+σtz. By a change of variables, Lsimple,t=Ex0,xt∼q[‖ϵθ(xt,t)−xt−α¯tx0σt‖2]=Ext∼q,x0∼q(⋅|xt)[‖ϵθ(xt,t)−xt−α¯tx0σt‖2] and the term inside becomes a least squares regression, so if the network actually reaches the global minimum of loss, then we have ϵθ(xt,t)=xt−α¯tEq[x0|xt]σt=−σt∇xtln⁡q(xt)

Thus, given a good score-based network, its predicted score is a good prediction of the noise (after scaling by σt), and thus can be used for denoising.

Conversely, the continuous limit xt−1=xt−dt,βt=β(t)dt,ztdt=dWt of the backward equation xt−1=xtαt−βtσtαtϵθ(xt,t)+βtzt;zt∼𝒩(0,I) gives us precisely the same equation as score-based diffusion: xt−dt=xt(1+β(t)dt/2)+β(t)∇xtln⁡q(xt)dt+β(t)dWtThus, at infinitesimal steps of DDPM, a denoising network performs score-based diffusion.

## Main variants

### Noise schedule

Illustration for a linear diffusion noise schedule. With settings β1=10−4,β1000=0.02.

In DDPM, the sequence of numbers 0=σ0<σ1<⋯<σT<1 is called a (discrete time) noise schedule. In general, consider a strictly increasing monotonic function σ of type ℝ->(0,1), such as the sigmoid function. In that case, a noise schedule is a sequence of real numbers λ1<λ2<⋯<λT. It then defines a sequence of noises σt:=σ(λt), which then derives the other quantities βt=1−1−σt21−σt−12.

In order to use arbitrary noise schedules, instead of training a noise prediction model ϵθ(xt,t), one trains ϵθ(xt,σt).

Similarly, for the noise conditional score network, instead of training fθ(xt,t), one trains fθ(xt,σt).

### Denoising Diffusion Implicit Model (DDIM)

The original DDPM method for generating images is slow, since the forward diffusion process usually takes T∼1000 to make the distribution of xT to appear close to Gaussian. However this means the backward diffusion process also take 1000 steps. Unlike the forward diffusion process, which can skip steps as xt|x0 is Gaussian for all t≥1, the backward diffusion process does not allow skipping steps. For example, to sample xt−2|xt−1∼𝒩(μθ(xt−1,t−1),Σθ(xt−1,t−1)) requires the model to first sample xt−1. Attempting to directly sample xt−2|xt would require us to marginalize out xt−1, which is generally intractable.

DDIM[22] is a method to take any model trained on DDPM loss, and use it to sample with some steps skipped, sacrificing an adjustable amount of quality. If we generate the Markovian chain case in DDPM to non-Markovian case, DDIM corresponds to the case that the reverse process has variance equals to 0. In other words, the reverse process (and also the forward process) is deterministic. When using fewer sampling steps, DDIM outperforms DDPM.

In detail, the DDIM sampling method is as follows. Start with the forward diffusion process xt=α¯tx0+σtϵ. Then, during the backward denoising process, given xt,ϵθ(xt,t), the original data is estimated as x0′=xt−σtϵθ(xt,t)α¯tthen the backward diffusion process can jump to any step 0≤s<t, and the next denoised sample is xs=α¯sx0′+σs2−(σ's)2ϵθ(xt,t)+σs′ϵwhere σs′ is an arbitrary real number within the range [0,σs], and ϵ∼𝒩(0,I) is a newly sampled Gaussian noise.[14] If all σs′=0, then the backward process becomes deterministic, and this special case of DDIM is also called "DDIM". The original paper noted that when the process is deterministic, samples generated with only 20 steps are already very similar to ones generated with 1000 steps on the high-level.

The original paper recommended defining a single "eta value" η∈[0,1], such that σs′=ησ~s. When η=1, this is the original DDPM. When η=0, this is the fully deterministic DDIM. For intermediate values, the process interpolates between them.

By the equivalence, the DDIM algorithm also applies for score-based diffusion models.

### Latent diffusion model (LDM)

Main article: Software:Latent diffusion model

Since the diffusion model is a general method for modelling probability distributions, if one wants to model a distribution over images, one can first encode the images into a lower-dimensional space by an encoder, then use a diffusion model to model the distribution over encoded images. Then to generate an image, one can sample from the diffusion model, then use a decoder to decode it into an image.[23]

The encoder-decoder pair is most often a variational autoencoder (VAE).

### Architectural improvements

[24][who?] proposed various architectural improvements. For example, they proposed log-space interpolation during backward sampling. Instead of sampling from xt−1∼𝒩(μ~t(xt,x~0),σ~t2I), they recommended sampling from 𝒩(μ~t(xt,x~0),(σtvσ~t1−v)2I) for a learned parameter v.

In the v-prediction formalism, the noising formula xt=α¯tx0+1−α¯tϵt is reparameterised by an angle ϕt such that cos⁡ϕt=α¯t and a "velocity" defined by cos⁡ϕtϵt−sin⁡ϕtx0. The network is trained to predict the velocity v^θ, and denoising is by xϕt−δ=cos⁡(δ)xϕt−sin⁡(δ)v^θ(xϕt).[25] This parameterization was found to improve performance, as the model can be trained to reach total noise (i.e. ϕt=90∘) and then reverse it, whereas the standard parameterization never reaches total noise since α¯t>0 is always true.[26]

### Classifier guidance

Classifier guidance was proposed in 2021 to improve class-conditional generation by using a classifier. The original publication used CLIP text encoders to improve text-conditional image generation.[27]

Suppose we wish to sample not from the entire distribution of images, but conditional on the image description. We don't want to sample a generic image, but an image that fits the description "black cat with red eyes". Generally, we want to sample from the distribution p(x|y), where x ranges over images, and y ranges over classes of images (a description "black cat with red eyes" is just a very detailed class, and a class "cat" is just a very vague description).

Taking the perspective of the noisy channel model, we can understand the process as follows: To generate an image x conditional on description y, we imagine that the requester really had in mind an image x, but the image is passed through a noisy channel and came out garbled, as y. Image generation is then nothing but inferring which x the requester had in mind.

In other words, conditional image generation is simply "translating from a textual language into a pictorial language". Then, as in noisy-channel model, we use Bayes theorem to get p(x|y)∝p(y|x)p(x) in other words, if we have a good model of the space of all images, and a good image-to-class translator, we get a class-to-image translator "for free". In the equation for backward diffusion, the score ∇ln⁡p(x) can be replaced by ∇xln⁡p(x|y)=∇xln⁡p(x)⏟score+∇xln⁡p(y|x)⏟classifier guidance where ∇xln⁡p(x) is the score function, trained as previously described, and ∇xln⁡p(y|x) is found by using a differentiable image classifier.

During the diffusion process, we need to condition on the time, giving∇xtln⁡p(xt|y,t)=∇xtln⁡p(y|xt,t)+∇xtln⁡p(xt|t)Although, usually the classifier model does not depend on time, in which case p(y|xt,t)=p(y|xt).

Classifier guidance is defined for the gradient of score function, thus for score-based diffusion network, but as previously noted, score-based diffusion models are equivalent to denoising models by ϵθ(xt,t)=−σt∇xtln⁡p(xt|t), and similarly, ϵθ(xt,y,t)=−σt∇xtln⁡p(xt|y,t). Therefore, classifier guidance works for denoising diffusion as well, using the modified noise prediction:[27]ϵθ(xt,y,t)=ϵθ(xt,t)−σt∇xtln⁡p(y|xt,t)⏟classifier guidance

#### With temperature

The classifier-guided diffusion model samples from p(x|y), which is concentrated around the maximum a posteriori estimate arg⁡maxxp(x|y). If we want to force the model to move towards the maximum likelihood estimate arg⁡maxxp(y|x), we can use pγ(x|y)∝p(y|x)γp(x) where γ>0 is interpretable as inverse temperature. In the context of diffusion models, it is usually called the guidance scale. A high γ would force the model to sample from a distribution concentrated around arg⁡maxxp(y|x). This sometimes improves quality of generated images.[27]

This gives a modification to the previous equation:∇xln⁡pβ(x|y)=∇xln⁡p(x)+γ∇xln⁡p(y|x)For denoising models, it corresponds to[28]ϵθ(xt,y,t)=ϵθ(xt,t)−γσt∇xtln⁡p(y|xt,t)

### Classifier-free guidance (CFG)

If we do not have a classifier p(y|x), we could still extract one out of the image model itself:[28] ∇xln⁡pγ(x|y)=(1−γ)∇xln⁡p(x)+γ∇xln⁡p(x|y) Such a model is usually trained by presenting it with both (x,y) and (x,None), allowing it to model both ∇xln⁡p(x|y) and ∇xln⁡p(x).

Note that for CFG, the diffusion model cannot be merely a generative model of the entire data distribution ∇xln⁡p(x). It must be a conditional generative model ∇xln⁡p(x|y). For example, in stable diffusion, the diffusion backbone takes as input both a noisy model xt, a time t, and a conditioning vector y (such as a vector encoding a text prompt), and produces a noise prediction ϵθ(xt,y,t).

For denoising models, it corresponds toϵθ(xt,y,t,γ)=ϵθ(xt,t)+γ(ϵθ(xt,y,t)−ϵθ(xt,t))As sampled by DDIM, the algorithm can be written as[29]ϵuncond<-ϵθ(xt,t)ϵcond<-ϵθ(xt,t,c)ϵCFG<-ϵuncond+γ(ϵcond−ϵuncond)x0<-(xt−σtϵCFG)/1−σt2xs<-1−σs2x0+σs2−(σs′)2ϵuncond+σs′ϵA similar technique applies to language model sampling. Also, if the unconditional generation ϵuncond<-ϵθ(xt,t) is replaced by ϵneg cond<-ϵθ(xt,t,c′), then it results in negative prompting, which pushes the generation away from c′ condition.[30][31]

### Samplers

Given a diffusion model, one may regard it either as a continuous process, and sample from it by integrating a SDE, or one can regard it as a discrete process, and sample from it by iterating the discrete steps. The choice of the "noise schedule" βt can also affect the quality of samples. A noise schedule is a function that sends a natural number to a noise level: t↦βt,t∈{1,2,…},β∈(0,1)A noise schedule is more often specified by a map t↦σt. The two definitions are equivalent, since βt=1−1−σt21−σt−12.

In the DDPM perspective, one can use the DDPM itself (with noise), or DDIM (with adjustable amount of noise). The case where one adds noise is sometimes called ancestral sampling.[32] One can interpolate between noise and no noise. The amount of noise is denoted η ("eta value") in the DDIM paper, with η=0 denoting no noise (as in deterministic DDIM), and η=1 denoting full noise (as in DDPM).

In the perspective of SDE, one can use any of the numerical integration methods, such as Euler–Maruyama method, Heun's method, linear multistep methods, etc. Just as in the discrete case, one can add an adjustable amount of noise during the integration.[33]

A survey and comparison of samplers in the context of image generation is in.[34]

### Other examples

Notable variants include[35] Poisson flow generative model,[36] consistency model,[37] critically damped Langevin diffusion,[38] GenPhys,[39] cold diffusion,[40] etc.

## Flow-based diffusion model

Abstractly speaking, the idea of diffusion model is to take an unknown probability distribution (the distribution of natural-looking images), then progressively convert it to a known probability distribution (standard Gaussian distribution), by building an absolutely continuous probability path connecting them. The probability path is in fact defined implicitly by the score function ∇ln⁡pt.

In denoising diffusion models, the forward process adds noise, and the backward process removes noise. Both the forward and backward processes are SDEs, though the forward process is integrable in closed-form, so it can be done at no computational cost. The backward process is not integrable in closed-form, so it must be integrated step-by-step by standard SDE solvers, which can be very expensive. The probability path in diffusions model is defined through an Itô process and one can retrieve the deterministic process by using the Probability ODE flow formulation.[1]

In flow-based diffusion models, the forward process is a deterministic flow along a time-dependent vector field, and the backward process is also a deterministic flow along the same vector field, but going backwards. Both processes are solutions to ODEs. If the vector field is well-behaved, the ODE will also be well-behaved.

Given two distributions π0 and π1, a flow-based model is a time-dependent velocity field vt(x) in [0,1]×ℝd, such that if we start by sampling a point x∼π0, and let it move according to the velocity field: ddtϕt(x)=vt(ϕt(x))t∈[0,1],starting from ϕ0(x)=x we end up with a point x1∼π1. The solution ϕt of the above ODE define a probability path pt=[ϕt]#π0 by the pushforward measure operator. In particular, [ϕ1]#π0=π1.

The probability path and the velocity field also satisfy the continuity equation, in the sense of probability distribution: ∂tpt+∇⋅(vtpt)=0 To construct a probability path, we start by construct a conditional probability path pt(x|z) and the corresponding conditional velocity field vt(x|z) on some conditional distribution q(z). A natural choice is the Gaussian conditional probability path: pt(x|z)=𝒩(mt(z),ζt2I) The conditional velocity field which corresponds to the geodesic path between conditional Gaussian path is vt(x|z)=ζt′ζt(x−mt(z))+mt′(z) The probability path and velocity field are then computed by marginalizing

pt(x)=∫pt(x|z)q(z)dz and vt(x)=𝔼q(z)[vt(x|z)pt(x|z)pt(x)]

### Optimal transport flow

The idea of optimal transport flow [41] is to construct a probability path minimizing the Wasserstein metric. The distribution on which we condition is an approximation of the optimal transport plan between π0 and π1: z=(x0,x1) and q(z)=Γ(π0,π1), where Γ is the optimal transport plan, which can be approximated by mini-batch optimal transport. If the batch size is not large, then the transport it computes can be very far from the true optimal transport.

### Rectified flow

The idea of rectified flow[42][43] is to learn a flow model such that the velocity is nearly constant along each flow path. This is beneficial, because we can integrate along such a vector field with very few steps. For example, if an ODE ϕt˙(x)=vt(ϕt(x)) follows perfectly straight paths, it simplifies to ϕt(x)=x0+t⋅v0(x0), allowing for exact solutions in one step. In practice, we cannot reach such perfection, but when the flow field is nearly so, we can take a few large steps instead of many little steps.

160px | 160px | 160px

|
|

Linear interpolation  | Rectified Flow  | Straightened Rectified Flow [1]

The general idea is to start with two distributions π0 and π1, then construct a flow field ϕ0={ϕt:t∈[0,1]} from it, then repeatedly apply a "reflow" operation to obtain successive flow fields ϕ1,ϕ2,…, each straighter than the previous one. When the flow field is straight enough for the application, we stop.

Generally, for any time-differentiable process ϕt, vt can be estimated by solving: minθ∫01𝔼x∼pt[‖vt(x,θ)−vt(x)‖2]dt.

In rectified flow, by injecting strong priors that intermediate trajectories are straight, it can achieve both theoretical relevance for optimal transport and computational efficiency, as ODEs with straight paths can be simulated precisely without time discretization.

Transport by rectified flow[42]

Specifically, rectified flow seeks to match an ODE with the marginal distributions of the linear interpolation between points from distributions π0 and π1. Given observations x0∼π0 and x1∼π1, the canonical linear interpolation xt=tx1+(1−t)x0,t∈[0,1] yields a trivial case x˙t=x1−x0, which cannot be causally simulated without x1. To address this, xt is "projected" into a space of causally simulatable ODEs, by minimizing the least squares loss with respect to the direction x1−x0: minθ∫01𝔼π0,π1,pt[‖(x1−x0)−vt(xt)‖2]dt.

The data pair (x0,x1) can be any coupling of π0 and π1, typically independent (i.e., (x0,x1)∼π0×π1) obtained by randomly combining observations from π0 and π1. This process ensures that the trajectories closely mirror the density map of xt trajectories but reroute at intersections to ensure causality.

The reflow process[42]

A distinctive aspect of rectified flow is its capability for "reflow", which straightens the trajectory of ODE paths. Denote the rectified flow ϕ0={ϕt:t∈[0,1]} induced from (x0,x1) as ϕ0=Rectflow((x0,x1)). Recursively applying this Rectflow(⋅) operator generates a series of rectified flows ϕk+1=Rectflow((ϕ0k(x0),ϕ1k(x1))). This "reflow" process not only reduces transport costs but also straightens the paths of rectified flows, making ϕk paths straighter with increasing k.

Rectified flow includes a nonlinear extension where linear interpolation xt is replaced with any time-differentiable curve that connects x0 and x1, given by xt=αtx1+βtx0. This framework encompasses DDIM and probability flow ODEs as special cases, with particular choices of αt and βt. However, in the case where the path of xt is not straight, the reflow process no longer ensures a reduction in convex transport costs, and also no longer straighten the paths of ϕt.[42]

These linear flow formulations were independently discovered and adapted from a different perspective for supervised inverse problems. For example, Inversion by Direct Iteration (InDI) formulates image restoration by learning a residual flow ODE that iteratively reverses a linear interpolation between a degraded input and a high-quality target to avoid regression-to-the-mean. InDI is effective for a variety of image restoration tasks.[44]

## Choice of architecture

Architecture of Stable Diffusion The denoising process used by Stable Diffusion

### Diffusion model

For generating images by DDPM, we need a neural network that takes a time t and a noisy image xt, and predicts a noise ϵθ(xt,t) from it. Since predicting the noise is the same as predicting the denoised image, then subtracting it from xt, denoising architectures tend to work well. For example, the U-Net, which was found to be good for denoising images, is often used for denoising diffusion models that generate images.[45]

For DDPM, the underlying architecture ("backbone") does not have to be a U-Net. It just has to predict the noise somehow. For example, the diffusion transformer (DiT) uses a Transformer to predict the mean and diagonal covariance of the noise, given the textual conditioning and the partially denoised image. It is the same as standard U-Net-based denoising diffusion model, with a Transformer replacing the U-Net.[46] Mixture of experts-Transformer can also be applied.[47]

DDPM can be used to model general data distributions, not just natural-looking images. For example, Human Motion Diffusion[48] models human motion trajectory by DDPM. Each human motion trajectory is a sequence of poses, represented by either joint rotations or positions. It uses a Transformer network to generate a less noisy trajectory out of a noisy one.

### Conditioning

The base diffusion model can only generate unconditionally from the whole distribution. For example, a diffusion model learned on ImageNet would generate images that look like a random image from ImageNet. To generate images from just one category, one would need to impose the condition, and then sample from the conditional distribution. Whatever condition one wants to impose, one needs to first convert the conditioning into a vector of floating point numbers, then feed it into the underlying diffusion model neural network. However, one has freedom in choosing how to convert the conditioning into a vector.

Stable Diffusion, for example, imposes conditioning in the form of cross-attention mechanism, where the query is an intermediate representation of the image in the U-Net, and both key and value are the conditioning vectors. The conditioning can be selectively applied to only parts of an image, and new kinds of conditionings can be finetuned upon the base model, as used in ControlNet.[49]

As a particularly simple example, consider image inpainting. The conditions are x~, the reference image, and m, the inpainting mask. The conditioning is imposed at each step of the backward diffusion process, by first sampling x~t∼N(α¯tx~,σt2I), a noisy version of x~, then replacing xt with (1−m)⊙xt+m⊙x~t, where ⊙ means elementwise multiplication.[50] Another application of cross-attention mechanism is prompt-to-prompt image editing.[51]

Conditioning is not limited to just generating images from a specific category, or according to a specific caption (as in text-to-image). For example,[48] demonstrated generating human motion, conditioned on an audio clip of human walking (allowing syncing motion to a soundtrack), or video of human running, or a text description of human motion, etc. For how conditional diffusion models are mathematically formulated, see a methodological summary in.[52]

### Upscaling

As generating an image takes a long time, one can try to generate a small image by a base diffusion model, then upscale it by other models. Upscaling can be done by GAN,[53] Transformer,[54] or signal processing methods like Lanczos resampling.

Diffusion models themselves can be used to perform upscaling. Cascading diffusion model stacks multiple diffusion models one after another, in the style of Progressive GAN. The lowest level is a standard diffusion model that generate 32x32 image, then the image would be upscaled by a diffusion model specifically trained for upscaling, and the process repeats.[45]

In more detail, the diffusion upscaler is trained as follows:[45]

* Sample (x0,z0,c), where x0 is the high-resolution image, z0 is the same image but scaled down to a low-resolution, and c is the conditioning, which can be the caption of the image, the class of the image, etc.
* Sample two white noises ϵx,ϵz, two time-steps tx,tz. Compute the noisy versions of the high-resolution and low-resolution images: {xtx=α¯txx0+σtxϵxztz=α¯tzz0+σtzϵz.
* Train the denoising network to predict ϵx given xtx,ztz,tx,tz,c. That is, apply gradient descent on θ on the L2 loss ‖ϵθ(xtx,ztz,tx,tz,c)−ϵx‖22.

## Examples

This section collects some notable diffusion models, and briefly describes their architecture.

### OpenAI

Main articles: Software:DALL-E and Sora (text-to-video model)

The DALL-E series by OpenAI are text-conditional diffusion models of images.

The first version of DALL-E (2021) is not actually a diffusion model. Instead, it uses a Transformer architecture that autoregressively generates a sequence of tokens, which is then converted to an image by the decoder of a discrete VAE. Released with DALL-E was the CLIP classifier, which was used by DALL-E to rank generated images according to how close the image fits the text.

GLIDE (2022–03)[55] is a 3.5-billion diffusion model, and a small version was released publicly.[5] Soon after, DALL-E 2 was released (2022–04).[56] DALL-E 2 is a 3.5-billion cascaded diffusion model that generates images from text by "inverting the CLIP image encoder", the technique which they termed "unCLIP".

The unCLIP method contains 4 models: a CLIP image encoder, a CLIP text encoder, an image decoder, and a "prior" model (which can be a diffusion model, or an autoregressive model). During training, the prior model is trained to convert CLIP image encodings to CLIP text encodings. The image decoder is trained to convert CLIP image encodings back to images. During inference, a text is converted by the CLIP text encoder to a vector, then it is converted by the prior model to an image encoding, then it is converted by the image decoder to an image.

Sora (2024–02) is a diffusion Transformer model (DiT).

### Lightricks

LTX is a family of open source artificial intelligence video foundation models developed by Lightricks, first released in November 2024.[57] The latest models, LTX-2, create videos based on user prompts, conditioned on text, images, video or audio - and capable of generating audio and visual in a unified way.[58][59] They were preceded by LTX Video, which was released in 2024 as the company's first text-to-video model.

### Stability AI

Main article: Software:Stable Diffusion

Stable Diffusion (2022–08), released by Stability AI, consists of a denoising latent diffusion model (860 million parameters), a VAE, and a text encoder. The denoising network is a U-Net, with cross-attention blocks to allow for conditional image generation.[60][23]

Stable Diffusion 3 (2024–03)[61] changed the latent diffusion model from the UNet to a Transformer model, and so it is a DiT. It uses rectified flow.

Stable Video 4D (2024–07)[62] is a latent diffusion model for videos of 3D objects.

### Google

Imagen (2022)[63][64] uses a T5-XXL language model to encode the input text into an embedding vector. It is a cascaded diffusion model with three sub-models. The first step denoises a white noise to a 64×64 image, conditional on the embedding vector of the text. This model has 2B parameters. The second step upscales the image by 64×64→256×256, conditional on embedding. This model has 650M parameters. The third step is similar, upscaling by 256×256→1024×1024. This model has 400M parameters. The three denoising networks are all U-Nets.

Muse (2023–01)[65] is not a diffusion model, but an encoder-only Transformer that is trained to predict masked image tokens from unmasked image tokens.

Imagen 2 (2023–12) is also diffusion-based. It can generate images based on a prompt that mixes images and text. No further information available.[66] Imagen 3 (2024–05) is too. No further information available.[67]

Veo (2024) generates videos by latent diffusion. The diffusion is conditioned on a vector that encodes both a text prompt and an image prompt.[68]

### Meta

Make-A-Video (2022) is a text-to-video diffusion model.[69][70]

CM3leon (2023) is not a diffusion model, but an autoregressive causally masked Transformer, with mostly the same architecture as LLaMa-2.[71][72]

Transfusion architectural diagram

Transfusion (2024) is a Transformer that combines autoregressive text generation and denoising diffusion. Specifically, it generates text autoregressively (with causal masking), and generates images by denoising multiple times over image tokens (with all-to-all attention).[73]

Movie Gen (2024) is a series of Diffusion Transformers operating on latent space and by flow matching.[74]

## See also

* Diffusion process
* Markov chain
* Variational inference
* Variational autoencoder

## Further reading

* Review papers
* Yang, Ling (2024-09-06), YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy, https://github.com/YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy, retrieved 2024-09-06
* Yang, Ling; Zhang, Zhilong; Song, Yang; Hong, Shenda; Xu, Runsheng; Zhao, Yue; Zhang, Wentao; Cui, Bin et al. (2023-11-09). "Diffusion Models: A Comprehensive Survey of Methods and Applications". ACM Comput. Surv. 56 (4): 105:1–105:39. doi:10.1145/3626235. ISSN 0360-0300. https://dl.acm.org/doi/abs/10.1145/3626235.
* Austin, Jacob; Johnson, Daniel D.; Ho, Jonathan; Tarlow, Daniel; Rianne van den Berg (2021). "Structured Denoising Diffusion Models in Discrete State-Spaces". arXiv:2107.03006 [cs.LG].
* Croitoru, Florinel-Alin; Hondru, Vlad; Ionescu, Radu Tudor; Shah, Mubarak (2023-09-01). "Diffusion Models in Vision: A Survey". IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (9): 10850–10869. doi:10.1109/TPAMI.2023.3261988. ISSN 0162-8828. PMID 37030794. Bibcode: 2023ITPAM..4510850C.
* Mathematical details omitted in the article.
* "Power of Diffusion Models" (in en). 2022-09-25. https://astralord.github.io/posts/power-of-diffusion-models/.
* Luo, Calvin (2022-08-25). "Understanding Diffusion Models: A Unified Perspective". arXiv:2208.11970 [cs.LG].
* Weng, Lilian (2021-07-11). "What are Diffusion Models?" (in en). https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
* Tutorials
* Nakkiran, Preetum; Bradley, Arwen; Zhou, Hattie; Advani, Madhu (2024). "Step-by-Step Diffusion: An Elementary Tutorial". arXiv:2406.08929 [cs.LG].
* "Guidance: a cheat code for diffusion models". 26 May 2022. https://benanne.github.io/2022/05/26/guidance.html. Overview of classifier guidance and classifier-free guidance, light on mathematical details.
* Catherine Higham, Desmond J. Higham, and Peter Grindrod: "Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicians", SIAM Review, Vol.67, No.3 (2025).

## References

1. ↑ 1.0 1.1 1.2 1.3 Song, Yang; Sohl-Dickstein, Jascha; Kingma, Diederik P.; Kumar, Abhishek; Ermon, Stefano; Poole, Ben (2021-02-10). "Score-Based Generative Modeling through Stochastic Differential Equations". arXiv:2011.13456 [cs.LG].
2. ↑ Croitoru, Florinel-Alin; Hondru, Vlad; Ionescu, Radu Tudor; Shah, Mubarak (2023). "Diffusion Models in Vision: A Survey". IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (9): 10850–10869. doi:10.1109/TPAMI.2023.3261988. PMID 37030794. Bibcode: 2023ITPAM..4510850C.
3. ↑ 3.0 3.1 Ho, Jonathan; Jain, Ajay; Abbeel, Pieter (2020). "Denoising Diffusion Probabilistic Models". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 33: 6840–6851. https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html.
4. ↑ Gu, Shuyang; Chen, Dong; Bao, Jianmin; Wen, Fang; Zhang, Bo; Chen, Dongdong; Yuan, Lu; Guo, Baining (2021). "Vector Quantized Diffusion Model for Text-to-Image Synthesis". arXiv:2111.14822 [cs.CV].
5. ↑ 5.0 5.1 GLIDE, OpenAI, 2023-09-22, https://github.com/openai/glide-text2im, retrieved 2023-09-24
6. ↑ Li, Yifan; Zhou, Kun; Zhao, Wayne Xin; Wen, Ji-Rong (August 2023). "Diffusion Models for Non-autoregressive Text Generation: A Survey". Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization. pp. 6692–6701. doi:10.24963/ijcai.2023/750. ISBN 978-1-956792-03-4. http://dx.doi.org/10.24963/ijcai.2023/750.
7. ↑ Xu, Weijie; Hu, Wenxiang; Wu, Fanyou; Sengamedu, Srinivasan (2023). "DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM". Findings of the Association for Computational Linguistics: EMNLP 2023 (Stroudsburg, PA, USA: Association for Computational Linguistics): 9040–9057. doi:10.18653/v1/2023.findings-emnlp.606. http://dx.doi.org/10.18653/v1/2023.findings-emnlp.606.
8. ↑ Zhang, Haopeng; Liu, Xiao; Zhang, Jiawei (2023). "DiffuSum: Generation Enhanced Extractive Summarization with Diffusion". Findings of the Association for Computational Linguistics: ACL 2023 (Stroudsburg, PA, USA: Association for Computational Linguistics): 13089–13100. doi:10.18653/v1/2023.findings-acl.828. http://dx.doi.org/10.18653/v1/2023.findings-acl.828.
9. ↑ Yang, Dongchao; Yu, Jianwei; Wang, Helin; Wang, Wen; Weng, Chao; Zou, Yuexian; Yu, Dong (2023). "Diffsound: Discrete Diffusion Model for Text-to-Sound Generation". IEEE/ACM Transactions on Audio, Speech, and Language Processing 31: 1720–1733. doi:10.1109/taslp.2023.3268730. ISSN 2329-9290. Bibcode: 2023ITASL..31.1720Y. http://dx.doi.org/10.1109/taslp.2023.3268730.
10. ↑ Janner, Michael; Du, Yilun; Tenenbaum, Joshua B.; Levine, Sergey (2022-12-20). "Planning with Diffusion for Flexible Behavior Synthesis". arXiv:2205.09991 [cs.LG].
11. ↑ Chi, Cheng; Xu, Zhenjia; Feng, Siyuan; Cousineau, Eric; Du, Yilun; Burchfiel, Benjamin; Tedrake, Russ; Song, Shuran (2024-03-14). "Diffusion Policy: Visuomotor Policy Learning via Action Diffusion". arXiv:2303.04137 [cs.RO].
12. ↑ Sohl-Dickstein, Jascha; Weiss, Eric; Maheswaranathan, Niru; Ganguli, Surya (2015-06-01). "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (in en). Proceedings of the 32nd International Conference on Machine Learning (PMLR) 37: 2256–2265. http://proceedings.mlr.press/v37/sohl-dickstein15.pdf.
13. ↑ Ho, Jonathan (Jun 20, 2020), hojonathanho/diffusion, https://github.com/hojonathanho/diffusion, retrieved 2024-09-07
14. ↑ 14.0 14.1 Weng, Lilian (2021-07-11). "What are Diffusion Models?" (in en). https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
15. ↑ "Generative Modeling by Estimating Gradients of the Data Distribution | Yang Song". https://yang-song.net/blog/2021/score/.
16. ↑ 16.0 16.1 Song, Yang; Ermon, Stefano (2019). "Generative Modeling by Estimating Gradients of the Data Distribution". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 32. https://proceedings.neurips.cc/paper/2019/hash/3001ef257407d5a371a96dcd947c7d93-Abstract.html.
17. ↑ Song, Yang; Sohl-Dickstein, Jascha; Kingma, Diederik P.; Kumar, Abhishek; Ermon, Stefano; Poole, Ben (2021-02-10). "Score-Based Generative Modeling through Stochastic Differential Equations". arXiv:2011.13456 [cs.LG].
18. ↑ ermongroup/ncsn, ermongroup, 2019, https://github.com/ermongroup/ncsn, retrieved 2024-09-07
19. ↑ "Sliced Score Matching: A Scalable Approach to Density and Score Estimation | Yang Song". https://yang-song.net/blog/2019/ssm/.
20. ↑ Anderson, Brian D.O. (May 1982). "Reverse-time diffusion equation models". Stochastic Processes and Their Applications 12 (3): 313–326. doi:10.1016/0304-4149(82)90051-5. ISSN 0304-4149. http://dx.doi.org/10.1016/0304-4149(82)90051-5.
21. ↑ Luo, Calvin (2022). "Understanding Diffusion Models: A Unified Perspective". arXiv:2208.11970v1 [cs.LG].
22. ↑ Song, Jiaming; Meng, Chenlin; Ermon, Stefano (3 Oct 2023). "Denoising Diffusion Implicit Models". arXiv:2010.02502 [cs.LG].
23. ↑ 23.0 23.1 Rombach, Robin; Blattmann, Andreas; Lorenz, Dominik; Esser, Patrick; Ommer, Björn (13 April 2022). "High-Resolution Image Synthesis With Latent Diffusion Models". arXiv:2112.10752 [cs.CV].
24. ↑ Nichol, Alexander Quinn; Dhariwal, Prafulla (2021-07-01). "Improved Denoising Diffusion Probabilistic Models" (in en). Proceedings of the 38th International Conference on Machine Learning (PMLR): 8162–8171. https://proceedings.mlr.press/v139/nichol21a.html.
25. ↑ Salimans, Tim; Ho, Jonathan (2021-10-06). "Progressive Distillation for Fast Sampling of Diffusion Models" (in en). The Tenth International Conference on Learning Representations (ICLR 2022). https://openreview.net/forum?id=TIdIXIpzhoI.
26. ↑ Lin, Shanchuan; Liu, Bingchen; Li, Jiashi; Yang, Xiao (2024). "Common Diffusion Noise Schedules and Sample Steps Are Flawed" (in en). IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 5404–5411. https://openaccess.thecvf.com/content/WACV2024/html/Lin_Common_Diffusion_Noise_Schedules_and_Sample_Steps_Are_Flawed_WACV_2024_paper.html.
27. ↑ 27.0 27.1 27.2 Dhariwal, Prafulla; Nichol, Alex (2021-06-01). "Diffusion Models Beat GANs on Image Synthesis". arXiv:2105.05233 [cs.LG].
28. ↑ 28.0 28.1 Ho, Jonathan; Salimans, Tim (2022-07-25). "Classifier-Free Diffusion Guidance". arXiv:2207.12598 [cs.LG].
29. ↑ Chung, Hyungjin; Kim, Jeongsol; Park, Geon Yeong; Nam, Hyelin; Ye, Jong Chul (2024-06-12). "CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models". arXiv:2406.08070 [cs.CV].
30. ↑ Sanchez, Guillaume; Fan, Honglu; Spangher, Alexander; Levi, Elad; Ammanamanchi, Pawan Sasanka; Biderman, Stella (2023-06-30). "Stay on topic with Classifier-Free Guidance". arXiv:2306.17806 [cs.CL].
31. ↑ Armandpour, Mohammadreza; Sadeghian, Ali; Zheng, Huangjie; Sadeghian, Amir; Zhou, Mingyuan (2023-04-26). "Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond". arXiv:2304.04968 [cs.CV].
32. ↑ Yang, Ling; Zhang, Zhilong; Song, Yang; Hong, Shenda; Xu, Runsheng; Zhao, Yue; Zhang, Wentao; Cui, Bin; Yang, Ming-Hsuan (2022). "Diffusion Models: A Comprehensive Survey of Methods and Applications". arXiv:2206.00364 [cs.CV].
33. ↑ Shi, Jiaxin; Han, Kehang; Wang, Zhe; Doucet, Arnaud; Titsias, Michalis K. (2024). "Simplified and Generalized Masked Diffusion for Discrete Data". arXiv:2406.04329 [cs.LG].
34. ↑ Karras, Tero; Aittala, Miika; Aila, Timo; Laine, Samuli (2022). "Elucidating the Design Space of Diffusion-Based Generative Models". arXiv:2206.00364v2 [cs.CV].
35. ↑ Cao, Hanqun; Tan, Cheng; Gao, Zhangyang; Xu, Yilun; Chen, Guangyong; Heng, Pheng-Ann; Li, Stan Z. (July 2024). "A Survey on Generative Diffusion Models". IEEE Transactions on Knowledge and Data Engineering 36 (7): 2814–2830. doi:10.1109/TKDE.2024.3361474. ISSN 1041-4347. Bibcode: 2024ITKDE..36.2814C.
36. ↑ Xu, Yilun; Liu, Ziming; Tian, Yonglong; Tong, Shangyuan; Tegmark, Max; Jaakkola, Tommi (2023-07-03). "PFGM++: Unlocking the Potential of Physics-Inspired Generative Models" (in en). Proceedings of the 40th International Conference on Machine Learning (PMLR): 38566–38591. https://proceedings.mlr.press/v202/xu23m.html.
37. ↑ Song, Yang; Dhariwal, Prafulla; Chen, Mark; Sutskever, Ilya (2023-07-03). "Consistency Models" (in en). Proceedings of the 40th International Conference on Machine Learning (PMLR): 32211–32252. https://proceedings.mlr.press/v202/song23a.
38. ↑ Dockhorn, Tim; Vahdat, Arash; Kreis, Karsten (2021-10-06). "Score-Based Generative Modeling with Critically-Damped Langevin Diffusion". arXiv:2112.07068 [stat.ML].
39. ↑ Liu, Ziming; Luo, Di; Xu, Yilun; Jaakkola, Tommi; Tegmark, Max (2023-04-05). "GenPhys: From Physical Processes to Generative Models". arXiv:2304.02637 [cs.LG].
40. ↑ Bansal, Arpit; Borgnia, Eitan; Chu, Hong-Min; Li, Jie; Kazemi, Hamid; Huang, Furong; Goldblum, Micah; Geiping, Jonas et al. (2023-12-15). "Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise" (in en). Advances in Neural Information Processing Systems 36: 41259–41282. https://proceedings.neurips.cc/paper_files/paper/2023/hash/80fe51a7d8d0c73ff7439c2a2554ed53-Abstract-Conference.html.
41. ↑ Tong, Alexander; Fatras, Kilian; Malkin, Nikolay; Huguet, Guillaume; Zhang, Yanlei; Rector-Brooks, Jarrid; Wolf, Guy; Bengio, Yoshua (2023-11-08). "Improving and generalizing flow-based generative models with minibatch optimal transport" (in en). Transactions on Machine Learning Research. ISSN 2835-8856. https://openreview.net/forum?id=CD9Snc73AW.
42. ↑ 42.0 42.1 42.2 42.3 Liu, Xingchao; Gong, Chengyue; Liu, Qiang (2022-09-07). "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow". arXiv:2209.03003 [cs.LG].
43. ↑ Liu, Qiang (2022-09-29). "Rectified Flow: A Marginal Preserving Approach to Optimal Transport". arXiv:2209.14577 [stat.ML].
44. ↑ Delbracio, Mauricio; Milanfar, Peyman (2023). "Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration". Transactions on Machine Learning Research. https://openreview.net/forum?id=VmyFF5lL3F.
45. ↑ 45.0 45.1 45.2 Ho, Jonathan; Saharia, Chitwan; Chan, William; Fleet, David J.; Norouzi, Mohammad; Salimans, Tim (2022-01-01). "Cascaded diffusion models for high fidelity image generation". The Journal of Machine Learning Research 23 (1): 47:2249–47:2281. ISSN 1532-4435. https://dl.acm.org/doi/abs/10.5555/3586589.3586636.
46. ↑ Peebles, William; Xie, Saining (March 2023). "Scalable Diffusion Models with Transformers". arXiv:2212.09748v2 [cs.CV].
47. ↑ Fei, Zhengcong; Fan, Mingyuan; Yu, Changqian; Li, Debang; Huang, Junshi (2024-07-16). "Scaling Diffusion Transformers to 16 Billion Parameters". arXiv:2407.11633 [cs.CV].
48. ↑ 48.0 48.1 Tevet, Guy; Raab, Sigal; Gordon, Brian; Shafir, Yonatan; Cohen-Or, Daniel; Bermano, Amit H. (2022). "Human Motion Diffusion Model". arXiv:2209.14916 [cs.CV].
49. ↑ Zhang, Lvmin; Rao, Anyi; Agrawala, Maneesh (2023). "Adding Conditional Control to Text-to-Image Diffusion Models". arXiv:2302.05543 [cs.CV].
50. ↑ Lugmayr, Andreas; Danelljan, Martin; Romero, Andres; Yu, Fisher; Timofte, Radu; Van Gool, Luc (2022). "RePaint: Inpainting Using Denoising Diffusion Probabilistic Models". arXiv:2201.09865v4 [cs.CV].
51. ↑ Hertz, Amir; Mokady, Ron; Tenenbaum, Jay; Aberman, Kfir; Pritch, Yael; Cohen-Or, Daniel (2022-08-02). "Prompt-to-Prompt Image Editing with Cross Attention Control". arXiv:2208.01626 [cs.CV].
52. ↑ Zhao, Zheng; Luo, Ziwei; Sjölund, Jens; Schön, Thomas (2025-06-19). "Conditional sampling within generative diffusion models" (in en). Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 383 (2299). doi:10.1098/rsta.2024.0329. ISSN 1364-503X. PMC 12177524. https://royalsocietypublishing.org/doi/10.1098/rsta.2024.0329.
53. ↑ Wang, Xintao; Xie, Liangbin; Dong, Chao; Shan, Ying (2021). "Real-ESRGAN: Training Real-World Blind Super-Resolution With Pure Synthetic Data" (in en). International Conference on Computer Vision. pp. 1905–1914. https://openaccess.thecvf.com/content/ICCV2021W/AIM/papers/Wang_Real-ESRGAN_Training_Real-World_Blind_Super-Resolution_With_Pure_Synthetic_Data_ICCVW_2021_paper.pdf.
54. ↑ Liang, Jingyun; Cao, Jiezhang; Sun, Guolei; Zhang, Kai; Van Gool, Luc; Timofte, Radu (2021). "SwinIR: Image Restoration Using Swin Transformer" (in en). International Conference on Computer Vision, 2021. pp. 1833–1844. https://openaccess.thecvf.com/content/ICCV2021W/AIM/papers/Liang_SwinIR_Image_Restoration_Using_Swin_Transformer_ICCVW_2021_paper.pdf.
55. ↑ Nichol, Alex; Dhariwal, Prafulla; Ramesh, Aditya; Shyam, Pranav; Mishkin, Pamela; McGrew, Bob; Sutskever, Ilya; Chen, Mark (2022-03-08). "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models". arXiv:2112.10741 [cs.CV].
56. ↑ Ramesh, Aditya; Dhariwal, Prafulla; Nichol, Alex; Chu, Casey; Chen, Mark (2022-04-12). "Hierarchical Text-Conditional Image Generation with CLIP Latents". arXiv:2204.06125 [cs.CV].
57. ↑ "Lightricks challenges AI giants with open-source text-to-video platform" (in en). 2024-11-22. https://www.calcalistech.com/ctechnews/article/hjshie0fjl.
58. ↑ Kemper, Jonathan (2026-01-11). "Lightricks open-sources AI video model LTX-2, challenges Sora and Veo" (in en-US). https://the-decoder.com/lightricks-open-sources-ai-video-model-ltx-2-challenges-sora-and-veo/.
59. ↑ Lightricks. "Lightricks Releases LTX-2: The First Complete Open-Source AI Video Foundation Model" (in en). https://www.prnewswire.com/news-releases/lightricks-releases-ltx-2-the-first-complete-open-source-ai-video-foundation-model-302593012.html.
60. ↑ Alammar, Jay. "The Illustrated Stable Diffusion". https://jalammar.github.io/illustrated-stable-diffusion/.
61. ↑ Esser, Patrick; Kulal, Sumith; Blattmann, Andreas; Entezari, Rahim; Müller, Jonas; Saini, Harry; Levi, Yam; Lorenz, Dominik; Sauer, Axel (2024-03-05). "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis". arXiv:2403.03206 [cs.CV].
62. ↑ Xie, Yiming; Yao, Chun-Han; Voleti, Vikram; Jiang, Huaizu; Jampani, Varun (2024-07-24). "SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency". arXiv:2407.17470 [cs.CV].
63. ↑ "Imagen: Text-to-Image Diffusion Models". https://imagen.research.google/.
64. ↑ Saharia, Chitwan; Chan, William; Saxena, Saurabh; Li, Lala; Whang, Jay; Denton, Emily L.; Ghasemipour, Kamyar; Gontijo Lopes, Raphael et al. (2022-12-06). "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding" (in en). Advances in Neural Information Processing Systems 35: 36479–36494. https://proceedings.neurips.cc/paper_files/paper/2022/hash/ec795aeadae0b7d230fa35cbaf04c041-Abstract-Conference.html.
65. ↑ Chang, Huiwen; Zhang, Han; Barber, Jarred; Maschinot, A. J.; Lezama, Jose; Jiang, Lu; Yang, Ming-Hsuan; Murphy, Kevin; Freeman, William T. (2023-01-02). "Muse: Text-To-Image Generation via Masked Generative Transformers". arXiv:2301.00704 [cs.CV].
66. ↑ "Imagen 2 - our most advanced text-to-image technology" (in en). https://deepmind.google/technologies/imagen-2/.
67. ↑ Imagen-Team-Google; Baldridge, Jason; Bauer, Jakob; Bhutani, Mukul; Brichtova, Nicole; Bunner, Andrew; Castrejon, Lluis; Chan, Kelvin et al. (2024-12-13), Imagen 3
68. ↑ "Veo" (in en). 2024-05-14. https://deepmind.google/technologies/veo/.
69. ↑ "Introducing Make-A-Video: An AI system that generates videos from text". https://ai.meta.com/blog/generative-ai-text-to-video/.
70. ↑ Singer, Uriel; Polyak, Adam; Hayes, Thomas; Yin, Xi; An, Jie; Zhang, Songyang; Hu, Qiyuan; Yang, Harry; Ashual, Oron (2022-09-29). "Make-A-Video: Text-to-Video Generation without Text-Video Data". arXiv:2209.14792 [cs.CV].
71. ↑ "Introducing CM3leon, a more efficient, state-of-the-art generative model for text and images". https://ai.meta.com/blog/generative-ai-text-images-cm3leon/.
72. ↑ Chameleon Team (2024-05-16). "Chameleon: Mixed-Modal Early-Fusion Foundation Models". arXiv:2405.09818 [cs.CL].
73. ↑ Zhou, Chunting; Yu, Lili; Babu, Arun; Tirumala, Kushal; Yasunaga, Michihiro; Shamis, Leonid; Kahn, Jacob; Ma, Xuezhe; Zettlemoyer, Luke (2024-08-20). "Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model". arXiv:2408.11039 [cs.AI].
74. ↑ Movie Gen: A Cast of Media Foundation Models, The Movie Gen team @ Meta, October 4, 2024.

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Diffusion model. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Diffusion_model&oldid=4416926"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template