Smoothing the Score Function to Enhance Generalization in Diffusion Models

Abstract

Diffusion models achieve remarkable generation quality, yet face a fundamental challenge known as memorization, where generated samples can replicate training samples exactly. We develop a theoretical framework to explain this phenomenon by showing that the empirical score function (the score function corresponding to the empirical distribution) is a weighted sum of the score functions of Gaussian distributions, in which the weights are sharp softmax functions. This structure causes individual training samples to dominate the score function, resulting in sampling collapse. In practice, approximating the empirical score function with a neural network can partially alleviate this issue and improve generalization. Our theoretical framework explains why: In training, the neural network learns a smoother approximation of the weighted sum, allowing the sampling process to be influenced by local manifolds rather than single points. Leveraging this insight, we propose two novel methods to further enhance generalization: (1) Noise Unconditioning enables each training sample to adaptively determine its score function weight to increase the effect of more training samples, thereby preventing single-point dominance and mitigating collapse. (2) Temperature Smoothing introduces an explicit parameter to control the smoothness. By increasing the temperature in the softmax weights, we naturally reduce the dominance of any single training sample and mitigate memorization. Experiments across multiple datasets validate our theoretical analysis and demonstrate the effectiveness of the proposed methods in improving generalization while maintaining high generation quality.

Authors 3

Xinyu Zhou, Jiawei Zhang, Stephen J. Wright

Specification

Official page

arxiv.org/abs/2601.19285

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Arxiv announce type

replace

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

arXiv id

2601.19285

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Categories

cs.LG

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

PDF

arxiv.org/pdf/2601.19285

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Primary category

cs.LG

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Published

11 Sept 2026

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance