FisherDistill

Abstract

Does refinement need a learned discriminator?

Recent advances in diffusion distillation have enabled high-quality image generation in only one or a few network function evaluations (NFEs). Despite their efficiency, aggressively distilled generators often retain a residual distribution gap from their multi-step teachers, motivating recent methods to introduce adversarial refinement with a learned discriminator. In this work, we ask whether such refinement necessarily requires a learned adversarial discriminator. We introduce FisherDistill, a simple alternative based on Fisher linear discriminant analysis. Using intermediate representations from the frozen teacher, we estimate the first- and second-order statistics of real and generated samples and derive a closed-form Fisher direction that maximally separates their feature distributions relative to within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective without requiring an additional adversarial discriminator or its optimization loop. Across multiple distillation frameworks, our approach consistently improves generation quality over their respective distilled baselines and substantially closes the gap to adversarially refined counterparts.

Four stacked histograms of Fisher projections of teacher features: Teacher, DMD2 without GAN, DMD2 with GAN, and FisherDistill, on a shared axis. — **The frozen teacher already sees the gap.** Held-out Fisher projections of EDM-teacher features on ImageNet-64: one-step DMD2 w/o GAN drifts away from the multi-step teacher, while the closed-form Fisher refinement brings it back. Dashed lines mark means.

Method

A closed-form critic in place of a GAN

Overview diagram: a few-step generator is trained with the base distillation loss; generated and real images are noised and passed through the frozen teacher, whose feature statistics define a closed-form Fisher direction that corrects the generator. — **Overview of FisherDistill.** We augment an existing distillation framework with Fisher refinement without modifying its base distillation objective. Real-image teacher features are precomputed offline, while generated-image features and their statistics are estimated online. Here, μ^r_ℓ, μ^g_ℓ and Σ^r_ℓ, Σ^g_ℓ denote the means and covariances of the real and generated pooled features, respectively. At each selected layer, the closed-form Fisher direction u*_ℓ = [ ½(Σ^r_ℓ + Σ^g_ℓ) + λ_ℓI ]⁻¹ (μ^r_ℓ − μ^g_ℓ) points from the generated toward the real feature mean, where λ_ℓ regularizes the covariance for numerical stability. The Fisher loss converts this direction into a corrective signal toward the real distribution, while gradients propagate through the frozen teacher to update only the generator. No learned discriminator or adversarial optimization is required.

Closed-form Fisher critic

The residual gap is visible in the frozen teacher’s features (see the figure next to the abstract), so we build the critic there. At a fixed noise level t_F, each sample is passed through the frozen teacher; for every selected layer ℓ the feature f_ℓ concatenates the per-channel spatial mean and standard deviation (or token statistics for DiTs).

Fisher direction

u*_ℓ = [ ½(Σ^r_ℓ + Σ^g_ℓ) + λ_ℓI ]⁻¹ (μ^r_ℓ − μ^g_ℓ)

Signed margin

r_ℓ(f) = f^⊤u*_ℓ − ½ u*_ℓ^⊤(μ^r_ℓ + μ^g_ℓ)

Closed-form critic & loss

D^F_ℓ(f) = σ(r_ℓ(f)), ℒ^F_ℓ = −𝔼[ log D^F_ℓ(f^g_ℓ) ]

Generator objective

ℒ_G = ℒ_distill + λ_F · (1/|S|) ∑_ℓ∈S ℒ^F_ℓ

A log-density ratioUnder a shared-covariance Gaussian model the margin is exactly the real-to-generated log-density ratio, so D^F is a closed-form counterpart of a learned discriminator.

Same gradient structureIts per-sample gradient mirrors adversarial refinement: a weight 1 − D^F, the diffusion scaling, and the Fisher direction pushed through the frozen teacher.

Lagged online statisticsReal statistics are fixed; generated moments are tracked with an EMA, and the critic is one step lagged so it never scores the features it was estimated from.

Where to read the teacher

The critic is only as useful as the layers it reads. Held-out separability varies strongly across teacher layers, and it predicts which layers work.

Held-out Fisher AUROC for real versus generated features across 38 EDM teacher layers, with a real-versus-real null floor; the three most separable and three near-null layers are marked. — (a) Held-out Fisher AUROC across frozen EDM teacher layers (ImageNet 64×64) for real-vs.-generated features, with a real-vs.-real null floor. Stars: three most separable layers; squares: three near-null layers.

FID during training for the most separable layers versus near-null layers and the no-critic continuation. — (b) The most separable layers reach FID 1.34; near-null layers reach only 1.69 and track the no-critic continuation.

Results

Closing the gap to adversarial refinement, without the adversary

Each panel keeps the base framework, initialisation and budget fixed and changes only the refinement term. Lower FID is better.

GAN refinementRealUIDFisherDistill

FID per setting, each panel on its own scale starting at 0. GAN refinement is DMD2, SiD²A, 1.x-Distill Stage II and SiD²A-DiT respectively; all baselines trained with the official code under a matched budget. Each group uses its base framework's evaluation protocol.

Full tables (precision/recall, CLIP, GenEval, aesthetics, preference)

Class-conditional ImageNet 64×64, EDM teacher

Method	NFE	FID ↓	Prec. ↑	Rec. ↑
EDM teacher (VP, SDE)	511	1.32
DMD-based
DMD	1	2.64	0.76	0.59
DMD2† (GAN)	1	1.42	0.74	0.60
RealUID†	1	3.07	0.74	0.58
FisherDistill	1	1.40	0.74	0.62
SiD-based
SiD (α = 1.2)	1	1.54	0.74	0.63
SiD²A† (GAN, α = 1.0)	1	1.31	0.74	0.62
RealUID†	1	1.45	0.74	0.62
FisherDistill	1	1.34	0.76	0.63

Text-to-image, zero-shot COCO-2014

Method	NFE	FID ↓	CLIP ↑	GenEval ↑	Aesth. ↑	HPSv2 ↑	ImgRwd ↑	PickScore ↑
Stable Diffusion v1.5 · DMD2
SD v1.5 teacher	50	8.60	0.310	0.442	5.466	0.267	0.096	20.61
DMD	1	9.28	0.302	0.347	5.333	0.255	−0.410	20.01
DMD2† (GAN)	1	8.58	0.300	0.343	5.414	0.256	−0.378	20.00
RealUID†	1	9.37	0.303	0.352	5.346	0.256	−0.332	20.06
FisherDistill	1	8.46	0.304	0.348	5.393	0.257	−0.350	20.07
SD3-Medium · 1.x-Distill
SD3-Medium teacher	28	23.39	0.330	0.708	5.370	0.294	1.027	22.50
1.x-Distill Stage I†	2	32.35	0.329	0.651	5.283	0.284	0.985	22.15
1.x-Distill Stage II† (GAN)	2	30.20	0.330	0.655	5.430	0.288	0.998	22.59
RealUID†	2	34.03	0.326	0.641	5.101	0.279	0.862	21.99
FisherDistill	2	29.54	0.330	0.682	5.483	0.294	1.059	22.37
SANA 0.6B · SiD-DiT
SANA 0.6B teacher	20	26.38	0.338	0.651	5.730	0.311	0.999	22.50
SiD-DiT	1	28.94	0.323	0.594	5.751	0.278	0.709	21.51
SiD²A-DiT† (GAN)	1	34.13	0.298	0.486	5.383	0.244	0.287	21.00
RealUID†	1	31.87	0.320	0.590	5.740	0.274	0.688	21.46
FisherDistill	1	25.24	0.326	0.597	5.744	0.286	0.820	21.80
SANA 1.6B · SiD-DiT
SANA 1.6B teacher	20	26.83	0.334	0.652	5.840	0.312	1.054	22.56
SiD-DiT	1	28.86	0.325	0.581	5.881	0.285	0.880	21.71
SiD²A-DiT† (GAN)	1	28.58	0.323	0.615	5.889	0.289	0.883	21.81
RealUID†	1	34.28	0.322	0.578	5.916	0.279	0.867	21.60
FisherDistill	1	26.73	0.326	0.603	5.816	0.289	0.884	21.73

† Reproduced with the official code under a matched training budget. Compare only within a group.

Samples

One step, side by side

Text-to-image

Side-by-side 512 by 512 samples: teacher, GAN-refined student and FisherDistill, for Stable Diffusion v1.5 on the left and SANA-1.6B on the right, with prompts. — 512×512 samples from our 1-step generators distilled from Stable Diffusion v1.5 (left, DMD2) and SANA-1.6B (right, SiD-DiT), next to the teacher and the GAN-refined student.

ImageNet 64×64, uncurated, one step

One sample per class for classes 0–99, identical noise across models.

DMD

SiD

10 by 10 grid of ImageNet 64x64 samples, one per class for classes 0 to 99. — FisherDistill + DMD (FID 1.40)

Citation

BibTeX

@article{bhosale2026fisherdistill,
  title   = {FisherDistill: Diffusion Distribution Distillation via Closed-Form Fisher Critics},
  author  = {Bhosale, Mahesh and Zhai, Yuanhao and Lyu, Siwei and Yuan, Junsong and Doermann, David and Chen, Defang},
  journal = {arXiv preprint},
  year    = {2026}
}

The arXiv identifier will be added once the preprint is live.