Stable Diffusion Sampling Methods: The Definitive Guide [Comparisons + Examples]

Cole B.
June 6, 2024

Today I am going to explain the stable diffusion sampling methods with comparisons to help you choose the best one.

Choosing the right sampling method can make or break the quality of your images, so picking the right one is important.

So if you want to learn how to choose the best sampling method, this guide is for you.

Let’s get started.

Stable Diffusion Sampling Methods Explained

Sampling methods play a crucial role in determining the quality and accuracy of the generated images by reducing noise in each sampling step.

Here’s how they impact the process:

Quality of Generated Images: Sampling methods influence how well the generated images match the target distribution. By effectively adding and removing noise during the sampling process, these methods help ensure that the final images are sharp, detailed, and realistic. Poor sampling methods can lead to blurry or unrealistic images.

Efficiency: Different sampling methods have varying levels of computational efficiency. Efficient methods can generate high-quality images faster, making the process more practical for real-time applications or large-scale image generation tasks.

Diversity of Outputs: Sampling methods affect the variety of images that can be produced. Good methods help explore the entire distribution of possible images, leading to a diverse set of outputs rather than producing similar or repetitive images.

Stability: Stability in sampling ensures that the process converges to a high-quality result without producing artifacts or noise. Stable methods maintain consistency across different runs, providing reliable and repeatable image generation.

In simpler terms, the sampling methods used in stable diffusion prompts and models help make sure that the images you create are clear, realistic, and varied.

They do this by carefully refining the images step by step, which also makes the process faster and more reliable.

Different methods can change how good the images look, how quickly they are made, and how different each image is from the others.

How Do Sampling Methods Work?

In simple terms, stable diffusion sampling methods are ways to create new data that look similar to a set of examples (training data).

Think of starting with a blurry image and then slowly making it clearer by removing the blur step by step. Each small change helps make the image look more like the original examples.

Different techniques can be used to make these small changes, and they help ensure that the final images or data are realistic and accurate.

A more in-depth explanation would be that stable diffusion sampling methods refer to a class of techniques used in probabilistic modeling, particularly within the context of generative models like diffusion models.

The diffusion process involves iteratively refining samples by adding and removing noise in a controlled manner, ultimately generating data that matches the target distribution.

Various algorithms, such as Langevin dynamics and Hamiltonian Monte Carlo, can be employed to implement stable diffusion sampling, each leveraging mathematical principles to ensure the stability and efficiency of the sampling process.

What Are The Types Of Sampling Methods For Stable Diffusion?

We are going to look at all the different types of sampling methods available in Automatic1111.

Euler

An Euler sampler in stable diffusion leverages the principles of Euler’s method to update and refine samples iteratively, transforming noise into realistic images.

While simple and efficient, it requires careful tuning of step sizes to balance between computational cost and the quality of the generated images.

Advantages and Limitations of Euler

Advantages:

Simplicity: Euler’s method is straightforward and easy to implement.
Efficiency: For small step sizes, it can be quite efficient in terms of computation.

Limitations:

Accuracy: Euler’s method can be less accurate compared to higher-order methods (e.g., Runge-Kutta methods), especially for larger step sizes.
Stability: It may require very small step sizes to maintain stability, which can increase the computational cost.

LMS

The LMS sampler in stable diffusion models is a sophisticated technique that enhances the accuracy, stability, and efficiency of the image generation process.

By leveraging information from multiple past steps, it refines noisy images iteratively, producing high-quality and realistic outputs.

Advantages of the LMS Sampling Method

Accuracy: By leveraging multiple past states, LMS sampling achieves higher accuracy compared to single-step methods, which rely solely on the current state. This results in more detailed and realistic images.
Stability: The use of multiple steps helps stabilize the sampling process, reducing the risk of divergence or artifacts in the generated images. This stability is particularly important for complex, high-dimensional data distributions.
Efficiency: Although LMS methods involve more calculations per step, they often converge faster to high-quality results, making them more efficient overall. The balance between predictor and corrector steps ensures that computational resources are used effectively.

Heun

The Heun sampler is a sophisticated numerical technique used in stable diffusion models to enhance the precision and stability of the sampling process, leading to higher-quality image generation.

The Heun sampler ensures that the noise is added and removed in a controlled, precise manner.

This results in clearer, more realistic images by accurately following the diffusion path through the latent space. The improved accuracy helps capture fine details and complex structures in the generated images, enhancing their overall quality.

Advantages of the Heun Sampler

Higher Accuracy: By averaging the slopes, Heun’s method provides a better approximation of the true solution, leading to higher accuracy in the generated samples.
Stability: The improved accuracy helps in maintaining the stability of the diffusion process, reducing the likelihood of numerical artifacts.
Efficiency: While more computationally intensive than the Euler method, Heun’s method strikes a good balance between complexity and performance, making it suitable for practical applications in image generation.

Restart

a Restart sampler periodically restarts the sampling process to enhance quality and diversity by avoiding local minima and exploring the sample space more effectively.

While not a standard term, the principles align with techniques used in iterative optimization and sampling methods.

Benefits of a Restart Sampler

Avoiding Local Minima: By restarting the sampling process, the sampler can escape from poor-quality local minima, leading to better overall samples.
Improved Diversity: Restarting allows the sampler to explore different regions of the sample space, resulting in a more diverse set of samples.
Enhanced Quality: Periodic restarts can lead to higher quality samples as the sampler is less likely to get stuck in suboptimal regions.

DDIM

A DDIM (Denoising Diffusion Implicit Model) sampler is an advanced method used in the context of stable diffusion models for generating high-quality images.

It builds upon the Denoising Diffusion Probabilistic Model (DDPM) framework, offering improvements in efficiency and flexibility. DDIM is designed to provide a more efficient and deterministic way to sample from a diffusion model.

Unlike traditional diffusion models that rely on a fully stochastic process to generate samples, DDIM introduces a more deterministic approach that can reduce the number of required sampling steps while maintaining or even improving the quality of the generated images.

Benefits of DDIM

Speed: The ability to reduce the number of denoising steps translates to faster image generation, which is particularly beneficial for real-time applications or when generating a large number of images.
Quality: Despite the reduced number of steps, DDIM maintains or even improves the quality of the generated images. The deterministic nature helps in producing consistent and sharp results.
Flexibility: The non-Markovian approach provides greater flexibility in the sampling process, allowing for fine-tuning of the trade-off between speed and quality.

PLMS

The PLMS sampler is based on linear multistep methods, which are numerical techniques used to solve differential equations.

In the context of diffusion models, these methods help in iteratively refining the noisy samples to produce clear and accurate images.

Advantages of PLMS Sampler

Higher Quality: The use of information from multiple steps results in more accurate and detailed images.
Faster Convergence: The efficiency of the multistep approach reduces the number of iterations needed, speeding up the sampling process.
Numerical Stability: The stability of the method ensures consistent results without unexpected artifacts or noise.

DPM & DPM++

A Denoising Probabilistic Model (DPM) sampler is a key component in the family of diffusion models, designed for generative modeling tasks such as image synthesis.

It operates by gradually transforming a simple initial distribution, typically Gaussian noise, into complex data distributions that match the target data (like images).

Advantages of DDPM Samplers

High-Quality Samples: DDPMs can generate high-quality images with fine details, as the iterative denoising process effectively captures the data distribution.
Stability: The gradual refinement process makes DDPMs more stable and less prone to mode collapse compared to some other generative models like GANs.
Flexibility: The framework is flexible and can be adapted to different types of data and various noise schedules.

Challenges

Computational Cost: The iterative nature of the denoising process makes DDPMs computationally expensive, as generating each sample requires many steps.

UniPC

The UniPC sampler is a sophisticated method that integrates multiple sampling techniques into a unified approach. It efficiently generates high-quality images by iteratively refining samples through a combination of stochastic and deterministic updates guided by gradient information.

Advantages of the UniPC Sampler

Efficiency: By combining stochastic and deterministic updates, the UniPC sampler achieves a balance between thorough exploration and precise refinement, leading to efficient sampling.
High Quality: The use of gradient-based refinement ensures that the generated samples are of high quality, with sharp details and realistic features.
Versatility: The unified framework allows the UniPC sampler to be applied to various types of data and tasks, making it a flexible tool in generative modeling.

What Are The Sampling Method's Schedule Types?

In diffusion models, a schedule type defines how noise is incrementally added to the data during the forward diffusion process (when data is transformed into pure noise) and how it is incrementally removed during the reverse process (when noise is transformed back into data).

The goal is to learn a denoising model that can progressively refine noisy data back to its original form.

Uniform

A uniform noise schedule in stable diffusion refers to the process of adding and removing noise to and from the data in a consistent, evenly spaced manner throughout the iterative steps of the diffusion process.

Advantages and Disadvantages of Uniform Noise Schedule

Advantages:

Simplicity: Easy to implement and understand.
Stability: Provides stable training by maintaining a consistent noise level throughout the process.

Disadvantages:

Suboptimal Performance: This may not always yield the best results compared to adaptive or non-uniform noise schedules that can focus more on critical steps of the diffusion process.
Uniform Complexity: Assumes that all steps in the diffusion process are equally important, which might not be the case for all data distributions.

Karras

the Karras noise schedule involves defining a function that specifies the noise level at each timestep. This function is then integrated into the diffusion model’s training loop, guiding the noise injection and removal process.

Exponential

The exponential noise schedule introduces noise in such a way that the noise level changes exponentially over the steps.

This means that at each step, the noise level is adjusted according to an exponential function, typically decreasing the noise rapidly at the beginning and more slowly towards the end.

Polyexpotential

A polyexponential noise schedule refers to a carefully designed sequence for adding and removing noise during the diffusion process.

By using a combination of different exponential rates, the polyexponential schedule can better adapt to the intricacies of the data distribution, improving the model’s ability to denoise and generate high-quality samples.

SGM Uniform

SGMs are a class of generative models that use the gradient (or score) of the data distribution to guide the sampling process. The score function is the gradient of the log probability density of the data.

During training, an SGM learns to estimate this score function for various noise levels added to the data.

Stable Diffusion Sampling Method Comparison

For this part, I am going to compare similar samplers using Automatic1111 to find the best-looking one.

I am going to use the same stable diffusion checkpoint, prompt, settings, and parameters, but I will show the results at many different sampling steps. These are my settings and parameters below:

Prompt: black and white cat, portrait photo, realistic, high quality
Seed: 2835483974
Checkpoint: Dreamshaper v8
CFG Scale: 7
Width and height: 512×512
Schedule Type: Automatic

DPM Samplers

As you can see, DPM fast needs a lot of steps (50-60+) in order to produce a good-quality image that would be comparable to DPM adaptive. Furthermore, DPM adaptive produces a high-quality image in very few steps.

DPM2 Samplers

Both DPM2 samplers have difficulty generating a good image at lower sampling steps. DPM2 a, in my opinion, creates a more realistic and better-looking image.

DPM++ Samplers

For DPM++, they all produce high-quality images at higher sampling steps. The main differences are in the details of the face, which some samplers like DPM++ 2S a and DPM++ SDE are slightly above (although DPM++ SDE has a bad chin).

Euler a vs Euler

Euler is a solid sampler. Although not the highest possible quality, it can still produce decent-looking images.

LMS + PLMS

PLMS is overall a less ideal option than LMS. It takes more steps in order to make a decent image while LMS makes a good image in about half the sampling steps.

Heun, Restart, DDIM, UniPC, LCM

Firstly, LCM is a highly volatile sampling method that produces wildly different images depending on the sampling steps. The rest of the sampling methods produce similar quality images with Restart and Heun being the best.

Best Sampling Method For Stable Diffusion

For good speed: DPM++ 2M.

For good quality: DPM++SDE.

I Want to Hear From You

Now that I have covered a bulk of the stable diffusion sampling methods.

I would like to hear from you.

Is there any sampling methods not on this list that you would like to see?

Let me know by leaving a comment below.