Abstract
Diffusion models typically suffer from error accumulation during iterative sampling, commonly referred to as exposure bias. We reveal systematic frequency-dependent discrepancies between training and inference, which can be interpreted as frequency-dependent SNR error. Crucially, the direction of this mismatch varies across models and timesteps, indicating that fixed correction rules do not generalize. We propose Spectral Alignment (SPA), a lightweight, guidance-based method that calibrates the power spectrum of intermediate predictions to a pre-computed prior. Our approach consists of two stages: (1) offline fitting of a parametric spectrum model from training data, and (2) inference-time guidance via efficient FFT-based gradient computation. SPA introduces minimal computational overhead (3-4%) and is complementary to Classifier-Free Guidance (CFG). We demonstrate consistent improvements across diverse architectures, from pixel-space models (DDPM, ADM) to latent diffusion models (SD2.0, SDXL) and flow-matching models (SD3.5, FLUX). Our implementation is available at https://github.com/SonyResearch/SPA.
1 Introduction
Diffusion models have become a dominant paradigm for image generation [ddpm2020, dhariwal2021diffusion, song2021denoising, rombach2022ldm, lipman2023flow]. Despite this rapid progress, a core challenge remains: during iterative sampling, small prediction errors can accumulate and lead to a distribution shift away from the target, a phenomenon often referred to as exposure bias. Prior studies have argued that exposure bias in diffusion sampling manifests primarily as a mismatch between the noise levels seen during training and those encountered at test time. Accordingly, these methods aim to correct the effective noise level to match the predefined noise schedule, e.g., by predicted-noise rescaling [ning2024elucidating], time-shifted sampling [li2024alleviating], and wavelet-based frequency reweighting [yu2025wavelet]. However, these approaches either assume frequency-uniform errors or rely on hand-designed correction rules and carefully tuned schedules.
In this work, we analyze the power spectrum of intermediate predictions during diffusion inference and reveal a systematic spectral mismatch between the forward process and inference trajectories (Figure 2). We interpret this discrepancy as frequency-dependent effective SNR errors that vary across models and timesteps, and show that correcting it leads to improved image quality. We discuss the details of this observation and its implications for method design in Section 2.
Based on this observation, we propose Spectral Alignment (SPA), a lightweight guidance-based calibration method applicable to a wide range of models. This method shifts the power spectra of intermediate predictions toward a precomputed timestep-dependent spectral prior. The overview of the method is shown in Figure 1. Our approach consists of two stages: (1) pre-calculation of the spectral prior and (2) sampling with guidance using spectral prior. In the first stage, a parametric spectrum model is fitted using training data. In the second stage, samples are steered toward the spectral prior via efficient FFT-based gradient computation.
The method can be applied to various models adaptively. We conducted experiments across a wide range of text-to-image diffusion models, from early architectures (DDPM [ddpm2020], ADM [dhariwal2021diffusion]) to LDMs (SD2.0 [sd20], SDXL [SDXL2024]) and state-of-the-art flow-matching models (SD3.5 [sd35], FLUX [flux]). We show that SPA outperforms other baseline methods, and consistently improves generation quality as measured by FID [TTUR] and reward models such as HPSv3 [Ma_2025_ICCV]. In addition, SPA introduces minimal computational overhead (3–4%) and can be easily integrated into existing generation pipelines.
-
We reveal that even modern flow-matching models exhibit spectral mismatch, and propose a calibration method using a precomputed reference power spectrum.
-
We demonstrate that correcting spectral mismatch with our approach improves generation quality with minimal computational overhead (3–4%).
-
We show that SPA is applicable to a wide variety of models without modification, from DDPM to state-of-the-art flow-matching models.
2 Motivation
In this section, we describe the key observations motivating the proposed method. We first introduce the notion of spectral mismatch from a frequency-dependent effective SNR error, and then present empirical observations that guided the design of our approach.
2.1 Preliminaries
2.1.1 Diffusion Models
The forward process of diffusion models gradually adds Gaussian noise to data according to a predefined noise schedule. The latent variable at an intermediate timestep, denoted by , can be obtained as:
| (1) |
where determines the noise level at timestep . The reverse process learns to denoise by predicting the noise using a neural network, given condition . The denoising step from to using DDIM [song2021denoising] is given by a deterministic update:
| (2) |
where is the predicted clean data estimated from using Tweedie’s formula:
| (3) |
2.1.2 Classifier-Free Guidance (CFG)
For conditional generation, CFG [ho2021cfg] computes a guided noise prediction by extrapolating between the conditional and unconditional estimates:
| (4) |
where is the guidance scale and denotes null conditioning. While CFG improves prompt adherence, high guidance scales often lead to oversaturation artifacts [wang2024analysis, sadat2025eliminating].
2.2 Spectral Mismatch
2.2.1 Frequency-Dependent SNR
Exposure bias in diffusion models refers to the mismatch between the distribution of intermediate variables observed during training (i.e., forward process) and those during inference (i.e., reverse process). Prior work suggests quantifying this mismatch through deviations from the predefined noise schedule, which is often referred to as an SNR error. Ning et al. [ning2024elucidating] proposed rescaling the predicted noise to correct such errors. Furthermore, Yu et al. [yu2025wavelet] showed that the mismatch differs between low- and high-frequency bands, indicating a frequency-dependent structure. In this work, we go beyond these perspectives and examine how the mismatch varies across the entire frequency spectrum, timesteps, and model choices, and how it can be corrected.
To analyze this, we define the signal-to-noise ratio at frequency and timestep as:
| (5) |
where and denote the Fourier coefficients at frequency . Since Gaussian noise has a flat power spectrum (i.e., is constant across frequencies), the frequency dependence of is primarily determined by the signal power . Accordingly, rather than directly estimating an SNR error, we focus on the mismatch in signal strength (power spectrum) between training and inference in this study.
During training, ground-truth images are available, and their frequency characteristics can be computed directly. During inference, however, is unknown, preventing a direct comparison of between the two settings. To address this, we use the estimated clean image as a common proxy in both cases. Specifically, we compute by applying a trained denoiser to sampled from the forward process, and from the reverse process at the corresponding timestep. By comparing the power spectra of these two estimates, we isolate the mismatch caused by the distributional gap between training-time and inference-time inputs, excluding the intrinsic high-frequency attenuation inherent to the posterior mean estimator. This allows us to quantify how the spectral mismatch varies across timesteps and models.
2.2.2 Empirical Observation
Figure 2(a) shows the averaged power spectra for each channel of and obtained from ADM and SDXL. We observe systematic spectral mismatches across all models tested, but crucially, the direction and pattern of the mismatch vary across models and timesteps. For instance, ADM [dhariwal2021diffusion] exhibits high-frequency attenuation, while Stable Diffusion 2.0 [sd20] shows low-frequency attenuation. More recent models such as SDXL [SDXL2024], SD3.5 [sd35], and FLUX [flux] show complex, channel-wise, time-dependent patterns. More examples can be found in the Appendix.
2.3 Motivation for the Proposed Method
A natural question is whether reducing the spectral mismatch directly improves sample quality, and if so, how to reduce it efficiently. In this paper, we propose a guidance-based method to reduce the spectral mismatch for two reasons. First, we aim to establish that spectral mismatch is a meaningful source of quality degradation and that correcting it yields measurable improvements. Second, we seek a practical, lightweight add-on applicable to existing pretrained models without retraining.
The model-dependent diversity shown in the observations above has two important implications. First, fixed correction rules (e.g., “boost high frequencies”) cannot generalize across architectures and denoising schedules. Second, the mismatch carries spatial structure that cannot be resolved by scalar variance correction alone. These observations motivate a data-driven approach that learns the appropriate target spectrum for each model.
3 Method
3.1 Overview
In this section, we describe our guidance-based spectral mismatch correction method. Our approach consists of two stages. In the first stage, we characterize the expected power spectrum of intermediate predictions by fitting a parametric model using training data and a pretrained diffusion model. This is an offline process required only once per model. In the second stage, during inference, we steer each denoising step toward the target spectrum via an efficient FFT-based gradient correction. We describe the details of the first stage in Section 3.2 and those of the second stage in Section 3.3. Figure 1 shows an overview of our method.
Remark.
This modeling process involves single-step model predictions (i.e., the forward process followed by one denoising step). While single-step prediction is not free of prediction error, it does not suffer from the error accumulation that arises during multi-step rollouts.
3.2 Target Power Spectrum Modeling
To define a correction target, we characterize the expected frequency properties of by fitting a parametric spectrum model from training data.
3.2.1 Power Spectrum Extraction
For each channel of , we compute the 2D power spectrum , where denotes the 2D DFT matrix, and denotes the element-wise squared magnitude. We then reduce this to a 1D representation via radial averaging, commonly referred to as the Radially Averaged Power Spectrum (RAPS) [RAPS]:
| (6) |
where is the normalized spatial frequency ( is the Nyquist frequency). All operations are applied independently to each channel.
3.2.2 Parametric Spectrum Model
It is well known that the power spectrum of natural images follows a power law [Field1987imagestat, VANDERSCHAAF1996]. We confirmed that , which is basically a blurry image, also follows the power law with slightly different parameters from natural images. We model the target spectrum as:
| (7) |
where , , , and are timestep-dependent parameters. The first term captures the power-law decay characteristic of natural images. The bias term and the linear term represent deviations from the pure power law. For the linear term in particular, we set by default and enable it only when it significantly improves the fit (e.g., SDXL, FLUX). All channels are modeled independently.
Model design.
It is technically possible to use a 2D representation directly; however, we chose to use RAPS for the following reasons. (1) A 1D power spectrum is easier to visualize, which makes analysis, explanation, and hyperparameter tuning much simpler, and (2) the regression model becomes much simpler in the 1D case, which makes the fitting process more stable and more robust.
3.2.3 Parameter Estimation Steps
Step 1: Per-timestep fitting.
For a discrete set of timesteps , we collect reference power spectra via the following procedure:
-
Sample clean images from the training dataset and apply the forward process to obtain .
-
Compute using Eq. (3) with the pretrained model.
-
Extract the RAPS via Eq. (6) and average across samples: .
-
Fit the parametric model (Eq. (7)) to via least-squares regression, obtaining for each channel and timestep independently.
Step 2: Temporal Interpolation.
To obtain smooth parameter functions over all timesteps , we apply cubic spline interpolation to each parameter:
| (8) |
where . Since the number of evaluation steps during inference is not fixed, the resulting model needs to capture the expected spectral characteristics of at any timestep.
3.3 Guidance-based Spectral Mismatch Calibration
Having established the target spectrum, we now describe how to steer the inference trajectory toward it. We frame this as a guidance problem, adapting Diffusion Posterior Sampling (DPS) [chung2023diffusion] to enforce spectral alignment at each denoising step. This inference-time approach serves as a practical add-on for existing pretrained models.
Guidance Loss.
At each reverse diffusion step, we compute via Tweedie’s formula (Eq. (3)), extract its RAPS , and compare against the target spectrum . The guidance loss is defined as:
| (9) |
where the quadratic loss is summed over frequency bins and averaged over channels. represents the center frequency of the -th bin. We use an unweighted MSE loss in log-spectrum space for simplicity and stability, and leave frequency-dependent weighting for future work.
Asymmetric Penalty.
During the fitting process, we observed that the sample-wise median of the spectrum from the forward process tends to be larger than the corresponding mean and exhibits positive skewness at each frequency . This indicates that –an observation during inference–falling too far below the target is undesirable, while exceeding the target is comparatively benign (see the Appendix for visualization).
This observation leads us to introduce an asymmetric penalty, which penalizes cases where the current spectrum falls below the target more strongly:
| (10) |
where controls the penalty intensity. The modified loss becomes:
| (11) |
Adaptation to LDMs.
The method we described above can be applied to latent diffusion models without modification. Our assumption so far is that the power spectrum of the intermediate variable follows a power law. VAEs of latent diffusion models are known to preserve the spatial information from the pixel space [rombach2022ldm]. In practice, the power spectrum of can be well fitted by our parametric power-law model. Visualization of the fitting can be found in the Appendix.
Overall Algorithm:
Algorithm 1 summarizes the complete sampling procedure with CFG. Our spectral alignment is applied after the CFG combination step (line 4) and before the standard denoising update (line 9). Thus, the method is generally compatible with most CFG variants [wang2024analysis, chung2025cfgplus, sadat2025eliminating, kwon2025tcfg] and other noise prediction modifications, since it operates on the combined prediction without any assumptions about how it was obtained.
Computational Efficiency.
The additional cost of our method is minimal. Spectrum extraction via FFT has complexity for -dimensional data, and radial averaging is . The gradient computation involves only elementwise operations and an inverse FFT for the backpropagation through the FFT, since it only involves the loss function, RAPS, and Tweedie’s formula (lines 5–7 of Algorithm 1). Because the gradient from is not used, no additional neural network evaluations or backpropagation steps are required for guidance. In practice, the per-step overhead is less than 5% of the total sampling time (see Section 4.2.3).
4 Experiments
4.1 Experimental Setup
Models.
We evaluate our method across a wide range of diffusion architectures: unconditional/class-conditional pixel-space models (DDPM [ddpm2020] on CelebA-HQ [karras2018progressive], ADM on ImageNet 256256 [imagenet]), text-to-image latent diffusion models (Stable Diffusion 2.0, SDXL [SDXL2024]), and flow-matching models (SD3.5 medium, FLUX.1 [dev]). For SDXL, we used the first stage without the refiner. We used the DDPM sampler for ADM, which is the default in the official implementation, and the DDIM sampler for the other models. We used 50 denoising steps for DDPM, 100 steps for ADM, and 30 steps for the other models.
Pre-calculation of target spectra.
We fit the parametric spectrum model (Section 3.2.2) using 10K samples. The model’s training dataset was used for DDPM and ADM (CelebA-HQ and ImageNet, respectively), and the LAION-Aesthetics V2 dataset [laion] was used for text-to-image models. The computational cost of this process is comparable to the scale of calculating metrics such as FID, and this process is only required once per model. We confirmed that the coefficient of determination ( score) exceeds 0.9 in most cases (see the Appendix for details).
Evaluation Metrics.
For unconditional and class-conditional models (DDPM, ADM), we report FID and KID as primary metrics, which reliably measure distributional fidelity in unconditional settings. We used clean-fid [clean-fid] to compute these metrics. We additionally report Density and Coverage [prdc], which are updated versions of Precision and Recall. For text-to-image models (SD2.0, SDXL, SD3.5, FLUX), we adopt HPSv3 and ImageReward (reward-model-based metrics) [Ma_2025_ICCV, xu2023imagereward] as primary indicators, since FID is known to correlate negatively with human preference for these models, especially at high CFG scales [SDXL2024]. We also report the CLIP score (ViT-G/14) as a measure of text alignment. For all evaluations, we use 5K prompts from the MS COCO validation set..
Baselines.
We compare against the following noise correction methods: (1) -rescaling [ning2024elucidating], which rescales the predicted noise to match the expected norm; (2) time-shift sampling [li2024alleviating], which adjusts the noise schedule at inference time; and (3) wavelet-based frequency regulation [yu2025wavelet], which applies frequency-band reweighting.
Hyperparameters were determined as follows. For (1), we used 1.004, following their ImageNet experiment. For SD2.0 and SDXL, we evaluated a candidate set and selected . For (2), the default values were used. The coefficient for (3) depends on the NFE and image resolution, which complicates the hyperparameter search. They reported the unimodality of FID with respect to the hyperparameters. Furthermore, since the qualitative effectiveness of this method becomes more prominent as the coefficient increases, we adopted the largest possible value within the range where no visual artifacts were observed. Specifically, we applied for ADM, for SD2.0, and for SDXL. Note that (1) has negligible computational overhead, (2) requires an additional timestep search (as detailed in Appendix F of their original work), and (3) involves a wavelet transform.
SPA Hyperparameters.
The hyperparameters used for our proposed method are as follows. DDPM: , ADM: , SD2.0 and SDXL: , SD3.5: , FLUX.1 [dev]: for CFG scale and for . Here, is the guidance strength and is the asymmetric penalty intensity. To manage computational costs, we first performed a preliminary qualitative assessment to identify promising parameter ranges, followed by a focused quantitative evaluation to finalize these values. Although SPA introduces two additional hyperparameters, they typically require tuning only once per model architecture. Notably, models with similar designs (e.g., SD2.0 and SDXL) converged to identical optimal values despite being searched independently, suggesting the robustness and transferability of these parameters.
4.2 Quantitative Results
4.2.1 Unconditional and Class-Conditional Generation
We evaluated our method on unconditional/class-conditional pixel-space models. Since all baselines are implemented for ADM, we conducted the comparison on ADM (ImageNet ). The results are shown in Table 1. On ADM, our method consistently outperforms the other methods. Density/Coverage are also improved, even though SPA guides intermediate variables toward the average spectrum. This indicates that SPA has benefits in terms of generation diversity. The time-shift sampler did not perform well in our experiments. This is likely due to the difference in image resolution, as they only experimented with resolutions up to .
We also observe consistent improvement on DDPM. However, the performance gain is slightly smaller than that on ADM, and simple rescaling works slightly better. This is likely because our method involves single-step model prediction in the target spectrum fitting stage. The target may be inaccurate when the single-step prediction itself is not accurate. We observed that the DDPM model did not perform well at predicting even from ground-truth (forward process) training data , which was not the case with the other models.
| Model | Method | FID | KID(×1000) | Density | Coverage |
|---|---|---|---|---|---|
| ADM (ImageNet) | Vanilla | 9.29 | 19.48 | 1.133 | 0.5939 |
| -rescaling | 8.00 | 11.01 | 1.146 | 0.5976 | |
| Time-shift | 15.35 | 77.47 | 0.761 | 0.4321 | |
| Wavelet reg. | 8.35 | 12.80 | 1.122 | 0.5929 | |
| SPA (Ours) | 7.81 | 10.25 | 1.226 | 0.6184 | |
| DDPM (CelebA) | Vanilla | 49.44 | 43.57 | 0.8186 | 0.0169 |
| -rescaling | 48.53 | 40.98 | 0.8791 | 0.0169 | |
| Time-shift | 90.19 | 46.65 | 0.8440 | 0.0038 | |
| Wavelet reg. | 49.46 | 43.31 | 0.8182 | 0.0168 | |
| SPA (Ours) | 48.63 | 41.36 | 0.8344 | 0.0167 |
| Model | Method | HPSv3 | ImageReward | CLIP Score |
|---|---|---|---|---|
| SD2.0 (=7.5) | Vanilla | 7.043 | 0.393 | 0.333 |
| -rescaling | 7.186 | 0.415 | 0.333 | |
| Time-shift | 7.101 | 0.407 | 0.333 | |
| Wavelet reg. | 7.193 | 0.429 | 0.332 | |
| SPA (Ours) | 7.238 | 0.421 | 0.333 | |
| SDXL (=7.5) | Vanilla | 8.426 | 0.791 | 0.341 |
| -rescaling | 8.528 | 0.800 | 0.340 | |
| Time-shift | 8.352 | 0.795 | 0.340 | |
| Wavelet reg. | 8.576 | 0.807 | 0.340 | |
| SPA (Ours) | 8.829 | 0.829 | 0.341 | |
| SD3.5 (=5) | Vanilla | 9.782 | 0.939 | 0.335 |
| SPA (Ours) | 9.975 | 0.976 | 0.334 | |
| FLUX.1 (=2.5) | Vanilla | 12.53 (11.55†) | 1.044 (0.989) | 0.328 (0.332) |
| SPA (Ours) | 12.57 (11.76) | 1.054 (1.005) | 0.328 (0.332) | |
| FLUX.1 (=3.5) | Vanilla | 12.47 (11.94) | 1.070 (1.044) | 0.330 (0.334) |
| SPA (Ours) | 12.48 (12.02) | 1.070 (1.052) | 0.330 (0.334) |
4.2.2 Text-to-Image Generation
Next, we evaluated text-to-image models. Although baseline methods were designed for ADM, we re-implemented them for SD2.0 and SDXL with minimal modifications. Table 2 shows the quantitative results.
SPA also performs well across various T2I models. Note that scores from reward models behave more like an ordinal scale than an interval or ratio scale. With that in mind, we conclude that SPA performs best among these baselines, especially on SDXL (approximately a gain). CLIP Scores appear to be saturated, but SPA does not degrade text alignment.
SPA also works well with both flow-matching models. For FLUX.1, SPA achieves decent performance with true CFG (the scores in parentheses), which is the standard two-pass CFG without distillation, but the score improvement with guidance distillation (without parentheses) is relatively small. In the following section, we discuss the results for FLUX.1 [dev] in detail.
Discussion on Guidance-Distilled Models.
FLUX.1 [dev] is a guidance-distilled model [meng2022on], trained to internalize the effect of CFG. We observe that SPA provides statistically significant improvements at lower guidance scales (, win-rate , ), while the effect diminishes at higher scales (). This suggests that guidance distillation may partially address spectral mismatch during training but does not fully eliminate it, particularly at lower guidance scales.
Notably, SPA shows stronger improvements on lower-quality samples. For images with HPSv3 scores in the bottom 20%, the win-rate increases to () even at . The average score gain for the bottom 5% of samples is () and (). This indicates that SPA serves as a useful refinement mechanism that selectively improves samples suffering from spectral distortion.
4.2.3 Computational Overhead
We evaluated the computational overhead of SPA on SDXL. We compared the time consumed for (A) the standard denoising step (lines 3–4 in Algorithm 1) with (B) the additional steps required for SPA (lines 5–8). The average time for (A) was 2.47 s and for (B) was 0.0952 s; thus, the overhead is only . Since the FFT is the bottleneck, SPA works efficiently with latent diffusion models. We observed a similar or smaller overhead on FLUX.1 [dev]. The relative overhead was only , since the FFT cost scales favorably with the latent spatial resolution. This confirms that the low overhead of SPA is not specific to SDXL.
4.3 Qualitative Results
Figure 3 shows results for SD2.0 and SDXL with the baseline methods listed in Section 4.1. In our experiments, SPA tends to refine flawed object shapes, as shown in the SD2.0 example in this figure. When the vanilla inference is already decent, SPA keeps the content as is, while slightly enhancing the color and detail, as seen in the SDXL sample.
Figure 4 shows results from FLUX.1 [dev]. This model has strong baseline performance; however, the model still occasionally generates unnatural structures. In such cases, SPA can refine the images. This behavior is practically useful for making adjustments after fixing the layout (via seed and prompts). Note that this is consistent with the statistical results in Section 4.2.2. We show five examples in the figure. The first two images contain defective objects (paddle/bat), while the remaining images have unnatural object structures.
Why Does SPA Improve Object Shapes?
Interestingly, SPA can refine object shapes in addition to statistical image properties (e.g., lighting, color distribution). We hypothesize two possible reasons for this phenomenon. (A) By fixing the SNR at each timestep, diffusion models can better utilize signal components from earlier steps, resulting in refined object shapes. (B) In latent diffusion models, VAEs are known to have disentangled channel representations (colors, object shape, layout, lighting, etc.) [sdxl_latent]; therefore, aligning these channels can alter the overall content.
5 Related Work
5.1 Exposure Bias
The concept of exposure bias was originally identified in autoregressive sequence generation, particularly in neural machine translation and language modeling [bengio2015scheduled, ranzato2016exposure]. In teacher forcing, models are trained to predict each token given ground-truth previous tokens, but at inference time, they must condition on their own predictions. This train-inference mismatch causes error accumulation, as models encounter inputs from a distribution different from that seen during training.
Diffusion models exhibit a structurally identical problem. Ning et al. [ning2024elucidating] explicitly identified this connection and proposed -rescaling to correct noise-level mismatch during inference. Li et al. [li2024error] analyzed error accumulation using an analytical model. Subsequent work has explored complementary corrections, as introduced in Section 4.1.
5.2 Guidance-based Correction Methods
Fundamentally, exposure bias stems from prediction errors in single-step predictions. Modifications to Classifier-Free Guidance (CFG) [ho2021cfg] aim to reduce such errors [wang2024analysis, chung2025cfgplus, kwon2025tcfg, sadat2025eliminating]. Although CFG is known as a truncation and error-correction method, it can also be a source of inference error, which often leads to oversaturation, unnatural contrast, and loss of fine details.
Discriminator Guidance [kim2023discriminator] refines model scores using a specially trained discriminator. Although it is a powerful way to refine predictions during inference, it requires unstable adversarial training and incurs additional inference costs for the discriminator at every denoising step. In the context of training-free guidance, MPGD [he2024manifold] proposes to alleviate manifold error using backpropagation through autoencoders, but this also requires additional inference costs. Note that our method is technically not mutually exclusive with these approaches, although empirical validation of their combination remains for future work.
6 Conclusion
We proposed Spectral Alignment (SPA) for diffusion models, which alleviates spectral mismatch and refines image quality during inference with minimal computational overhead. We demonstrated the effectiveness of SPA across various models, from classic pixel-space models to modern flow-matching models. In addition, the concept of spectral mismatch that we introduced can serve as a lens for further improvements in model training, where inspection tools are valuable.
Limitations.
The current formulation of our method assumes a single target spectrum as a prior; however, the optimal spectral prior may vary depending on the target data distribution. For example, illustrations and medical images exhibit different frequency properties. For conditional generation, it may be possible to use different target spectra depending on the conditioning information. In this paper, we did not explore alternative guidance methods or loss weighting strategies. Although we adopted a simple DPS-based guidance approach, there is room to explore more sophisticated methods. Regarding loss weighting, one could design a weighting schedule for across frequency bands and timesteps.
7 Acknowledgements
We would like to thank Naoki Murata for carefully reading the manuscript and giving us valuable feedback.
References
[Appendix] Spectral Prior for Reducing Exposure Bias in Diffusion Models
Appendix 0.A Details of the Regression of the Spectrum Model
For the regression of our parametric spectrum model, we used scipy.curve_fit in log space. The model is shown in Eq. (11) in the main document. The average coefficients of determination ( scores) across timesteps are shown in Table A1.
| Models | DDPM | ADM | SD2.0 | SDXL | SD3.5 | FLUX.1 [dev] |
|---|---|---|---|---|---|---|
| 0.927 | 0.948 | 0.999 | 0.998 | 0.998 | 0.953 |
Our spectrum model exhibits good fit () for all models, including latent diffusion models. Fig. A3 shows examples of fitting results on SD2.0. As we mentioned in the main document (Section 4.1), the spectrum model also works well for of latent diffusion models, as shown in this figure. Our spectrum model is based on the well-known property that the power spectrum of natural images follows a power law. This enables us to apply the same spectrum model to latent variables that share similar statistical properties to natural images.
Although we computed 10K samples for the regression, we still see fluctuations in the measured spectra. We obtain a smooth function by introducing the fitting step, which is expected to eliminate fluctuations and stabilize the following guidance phase.
Fig. A3 shows interpolation results of the parameters of the spectrum model: . The parameters are smoothly interpolated over timesteps, which enables us to use an arbitrary number of denoising steps.
Appendix 0.B Additional samples of Power Spectra During Inference
We show additional spectra from Stable Diffusion 2.0 and 3.5 in Figure A4 (in addition to Figure 2 of the main document). The timesteps are not exactly the same between the models, because SD3.5 uses uneven timesteps by default. We used the closest timesteps instead. The errors from these models also behave differently from the other models.
Appendix 0.C Analysis of the Hyperparameters
In this section, we show the effect of changing the hyperparameters. Our method introduced two hyperparameters: the guidance strength and penalty intensity (see Algorithm 1 in the main document).
0.C.1 Analysis on the guidance strength
We show the effect of guidance strength on FID and KID on DDPM in Figure A5. It basically shows unimodality to the evaluation metrics, which makes hyperparameter selection straightforward. We set the penalty intensity (equivalent to ablating the penalty function, Equation (10)) in this figure, since it is the optimal value for DDPM, and we can separate the effect of and .
0.C.2 Analysis on the penalty intensity
In this section, we analyze the effect of the penalty intensity . We show the results with SDXL to also report the weak interaction between and the CFG scale. We show the plot of HPSv3 vs in Figure A6 and qualitative results in Figure A8. We kept , which is the optimal value for SDXL. Note that, unlike the previous section, we can not set , since the penalty function is part of the guidance.
Quantitative Results
Figure A6 shows quantitative results of HPSv3 with different values and CFG scales. The penalty intensity enhances the score, as shown in this figure. The improvement in the scores gradually decreases as the parameter increases.
Skewness of a Spectrum
As we discussed in Section 3.3 of the main document, we designed our asymmetric penalty function due to the skewness of the measured spectra. We show examples of the histograms in Figure A7. Most of the measured spectra show positive skewness and “”, especially in . As we see in this figure, skewness varies across timesteps and frequency bands. Therefore, there is room for changing the penalty intensity depending on timesteps and frequency bands, which we leave for future work. Furthermore, estimating the optimal from skewness could also be an interesting future direction.
Qualitative Results
Figure A8 shows qualitative results with different penalty intensity on SDXL. Larger (i.e., stronger penalty) leads to better object shapes, more pronounced colors, and greater sharpness/contrast.
Appendix 0.D Implementation Details of the Baseline Methods
We adapted the three baseline methods for latent diffusion models with minimal modification. Please refer to Section 4.1 baselines in the main document. For -rescaling, we applied the scaling to the model prediction after CFG. For time-shift sampler and wavelet regulation, we applied each method to the latent variable before the sampler step ( in line 9, Algorithm 1).
Appendix 0.E Additional Qualitative Results
Figure A9, A10, and A11 show additional qualitative samples from text-to-image models. Please zoom in for details. We use four prompts from the MS COCO dataset for all the models. (1) “A kitchen counter with a rounded edge and shelves” (2) “A woman walking across a street in short shorts”, (3) “A white bathroom sink sitting under a mirror”, and (4) “A very tall brick building with bricked up windows”.
SPA enhances the original images in most samples and rarely affects them negatively.
Figure 10(b) shows results using the “dedistilled” checkpoint of FLUX.1[dev] (nyanko7/flux-dev-de-distill). This checkpoint re-introduced the standard two-pass CFG by deleting the weight embedding used for guidance distillation and is finetuned with some portion of blank conditioning. The difference made by SPA tends to be larger with this checkpoint than the official guidance-distilled version. Figure A11 shows results with styles other than the standard photo-real style. Although SPA guides samples toward the single target spectrum (spectrum prior), it does not negatively affect sampling of artistic styles.
Base prompt: beautiful city with canals
Style prompts: (1) “Impressionist painting, loose brushstrokes, light-focused color palette”, (2) “Cyberpunk aesthetic, neon lights, urban night, futuristic cityscape”, (3) “Vaporwave aesthetic, pastel neon, retro internet nostalgia”, and (4) “elegant brush strokes, delicate gray tones, large white space, washi paper texture, Zen” (numbering in a z-shaped)