Why sigma_data = 0.5 in EDM?

sienna223 · October 25, 2023, 2:13am

Hi, in Karras‘s paper Elucidating the Design Space of Diffusion-Based Generative Models, they specify sigma_{data} as 0.5, where is number come from?

github.com

huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_consistency_models.py#L58


      
          This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic
          methods the library implements for all schedulers such as loading and saving.
          
          Args:
              num_train_timesteps (`int`, defaults to 40):
                  The number of diffusion steps to train the model.
              sigma_min (`float`, defaults to 0.002):
                  Minimum noise magnitude in the sigma schedule. Defaults to 0.002 from the original implementation.
              sigma_max (`float`, defaults to 80.0):
                  Maximum noise magnitude in the sigma schedule. Defaults to 80.0 from the original implementation.
              sigma_data (`float`, defaults to 0.5):
                  The standard deviation of the data distribution from the EDM
                  [paper](https://huggingface.co/papers/2206.00364). Defaults to 0.5 from the original implementation.
              s_noise (`float`, defaults to 1.0):
                  The amount of additional noise to counteract loss of detail during sampling. A reasonable range is [1.000,
                  1.011]. Defaults to 1.0 from the original implementation.
              rho (`float`, defaults to 7.0):
                  The parameter for calculating the Karras sigma schedule from the EDM
                  [paper](https://huggingface.co/papers/2206.00364). Defaults to 7.0 from the original implementation.
              clip_denoised (`bool`, defaults to `True`):
                  Whether to clip the denoised outputs to `(-1, 1)`.

j3698 · January 12, 2024, 7:24am

That’s the standard deviation of the data. If you look at a more recent paper [2312.02696] Analyzing and Improving the Training Dynamics of Diffusion Models, that paper actually uses different data, so they could have used a different sigma_data; but instead they normalize the data to have standard deviation 0.5 so that they don’t have to change the other hyperparameters of EDM.

Topic		Replies	Views
DDIM v prediction problem 🧨 Diffusers	1	3135	September 5, 2023
Difference between origin EDM and diffusers EDM 🧨 Diffusers	1	617	December 4, 2023
Diffusers documentation has some error code,i have fixed it 🧨 Diffusers	1	413	May 26, 2023
What does "0.18215" mean in blog Stable Diffusion with 🧨 Diffusers? 🧨 Diffusers	3	5890	December 19, 2022
DDIM Inversion when setting prediction_type = "sample" 🧨 Diffusers	0	635	March 5, 2024

Why sigma_data = 0.5 in EDM?

Related topics