Paper Reading: DIRE for Diffusion-Generated Image Detection

Paper Link

Background

The existing detectors struggle to detect images generated by difussion models.

Diffusion models for image generation

  • DDPMs, ADM, PNDMs
  • text to image: VQ-difuusion, LDM, Stable Diffusion (V1 and V2)

Generated Image Detection

  • detect the GANs-generated images: spatial artifacts, frequency domain
  • detect the Diffusion-generated images: in detailed task

Method

DIRE(DIffusion Reconstruction Error):

DIRE measures the error between an input image and its reconstruction by a pre-trained diffusion model, because diffusion-generated images can be more approximately reconstructed by a pre-trained diffusion model compared to real images.

The images generated by diffusion models are essentially sampled from distribution of the diffusion generation space ($p_g(x)$) which is not exactly the same as ($p_r(x)$)
—-> samples from the diffusion generation space ($p_g(x)$) are more likely to be reconstructed by a pre-trained diffusion model while real images cannot.

Process:
$x_0$ (input image) —DDIM inversion (T steps)—> $x_T$ in isotropic Gaussian noise distribution —DDIM Reconstruction (T steps)—> $x_0’$ (recovered image) —> compute the difference between $x_0$ and $x_0’$ by $DIRE(x_0)=|x_0 - \mathbf{R}(\mathbf{I}(x_0))|$, where $\mathbf{R}(\mathbf{I}(x_0))$ is the recovered image $x_0’$ —> Use a simple binary classifier to distinguish DIRE between real image and diffusion-generated image.

DiffusionForensics: Dataset

source: LSUNBedroom and ImageNet models: 4 unconditional diffusion models, 4 text2image diffusion models (with “A photo of bedroom” text prompt for LSUNBedroom, “A photo of {class}” for {class} in ImageNet)

Evaluation/Experments

p1

Generation model is used to generated synthetic images. Reconstruction model is used to compute DIRE.

The existing method get a significant improvement on images generated by the same diffusion models as used in training, but still perform unsatisfactorily facing unseen diffusion models.

Cross-dataset: training the detector with images generated by models pre-trained on LSUN-Bedroom and then testing it on images produced by models pre-trained on ImageNet

Unseen GAN generation evaluation: Although the classifier never encounters any GAN-generated image during training, it achieves surprising performance when detecting GAN-generated images

Unseen Perturbations: Gaussian blur and JPEG compression. DIRE gets a perfect performance without performance drop

Why DIRE works:

p2

B.More Explanation of DIRE appendix in the paper: deviation caused by the approximation actually leads to our key idea of DIRE

Reading Summary

What is the contribution/novelty?

  • Proposed a new detector (DIRE) to detect Diffusion-generated image, and the generalization ability is impressive.
  • build a dataset for diffusion-generated image detection: DiffusionForensics
  • The accuracy is almost perfect, and the experiments are complete.

What is the existing issue?

  • DIRE works well on Cross-dataset testing, how to design a more challenging situation:
    • use a dataset that the difference of distribution is more obvious in generation model like a art dataset?
  • Can DIRE be used/expanded to detect other diffusion model based task? (like image editing(even detect which part is edited))
  • The reconstruction model(DDIM) of DIRE applies for 20 steps in total, what is the detection FPS of the method? Can we propose a faster method?