AI-generated image detection benchmarks: GenImage and its predecessors

This page sits in the deepfake detection hub beside the page on how AI image detection works. It covers whole-image benchmarks; the face-only sets are compared on the public deepfake datasets table.

01What is a generated-image detection benchmark?

A generated-image detection benchmark is a fixed set of real photographs and generated images, split into training and test parts, on which a binary classifier is scored. Three choices define it: where the real images come from, which generators made the fakes, and what the test images have been through (resizing, compression, cropping). A detector's score on the benchmark is a statement about those three choices and nothing else.

The benchmarks on this page differ from the face deepfake sets in one way that matters for reading them. FaceForensics++, Celeb-DF and the Deepfake Detection Challenge alter a real face inside a real frame, so the detector looks for the seam. GenImage and its predecessors contain images made from nothing but a label or a prompt, so there is no seam, and the detector has to learn the generator's own statistical trace. The page on fully generated faces covers the face-only case; this page covers general content.

02The 2020 ProGAN set: eleven generators, one training source

Wang, Wang, Zhang, Owens and Efros (CVPR 2020) built the first widely used set. Its training part uses ProGAN as its only generator: 20 ProGAN models, each trained on one LSUN object category, each contributing 36,000 training and 200 validation images with real and generated images in equal numbers, for 720,000 training and 4,000 validation images in all (arXiv 1912.11035). The test part holds images from 11 CNN generators: ProGAN, StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN, a DeepFakes face set, cascaded refinement networks, implicit maximum likelihood estimation, a super-resolution network and a low-light enhancement network.

The result that made the paper's title was generalisation. A standard ResNet-50 trained on the ProGAN images alone, with blur and JPEG applied to 10 percent of training images, reached a mean average precision of 92.6 across the 11 generators, with 99.6 on StyleGAN, 84.5 on BigGAN and 89.0 on the DeepFakes set (Table 2). Chance is 50 on that scale. Without augmentation the mean was 90.1. The authors read this as evidence that CNN generators of that period shared a common flaw, and the follow-up work on frequency artefacts by Frank et al. (ICML 2020) named it: upsampling leaves a spectral pattern across GAN architectures (arXiv 2003.08685).

03GenImage: eight generators, 2.68 million images

GenImage (Zhu et al., 2023) is the largest public set in the table below. It takes every real image in ImageNet, 1,281,167 for training and 50,000 for testing, and pairs them with 1,350,000 generated images made from the same 1,000 class labels, 2,681,167 images in all (arXiv 2306.08571). The eight generators are BigGAN (the only GAN), ADM, GLIDE, VQDM, Stable Diffusion V1.4, Stable Diffusion V1.5, Wukong and one commercial text-to-image service at its version 5.

The paper defines two tasks. Cross-generator classification trains on one generator's subset and tests on all eight. Degraded classification tests on low-resolution, blurred and compressed copies. Table 2 gives the cross-generator matrix for a ResNet-50: every same-generator cell is above 98.5 percent accuracy, the two Stable Diffusion cells reach 99.9 percent, and the model trained on Stable Diffusion V1.4 scores 54.9 percent on the commercial generator. Row averages run from 58.6 percent (trained on BigGAN) to 72.1 percent (trained on Stable Diffusion V1.4). The authors' reading is that a classifier trained on one generator "is likely to be tied to this generator and will not perform well on unknown generators".

Counts and generator families for the first four rows are from GenImage's Table 1 (the ProGAN row is listed there under its code name CNNSpot); the last row is from Deepfake-Eval-2024's Table 1, which reports the split's total but not its real-to-fake ratio.

04How do detectors score across GenImage's generators?

GenImage's Table 3 trains seven methods on Stable Diffusion V1.4 and averages their accuracy over the eight test subsets. The plain backbones lead: Swin-T 74.8 percent, ResNet-50 72.1, DeiT-S 71.6. The purpose-built detectors trail: GramNet 69.9, Spec 68.8, F3Net 68.7 and CNNSpot, the Wang et al. classifier, 64.2. Table 4 repeats the exercise with eight models trained on eight generators and averages all 64 results: Swin-T 70.3 percent, DeiT-S 67.6, ResNet-50 66.9, CNNSpot 61.7.

Two readings follow. First, a classifier that reached 92.6 mean average precision across GANs in 2020 averages 61.7 to 64.2 percent accuracy across the 2023 diffusion generators, which is the cleanest published measure of how far generator change moved the problem. Second, the authors note that "current CNN-based models like ResNet perform similarly to Transformer-based models", so the backbone is not where the gap closes. The glossary of detection terms defines cross-generator accuracy alongside the cross-dataset AUC used for face video.

05What happens under compression and resizing?

Corvi, Cozzolino, Zingarini, Poggi, Nagano and Verdoliva (2022) tested four detectors, all trained on ProGAN only, on 12 generators including StyleGAN2, StyleGAN3, BigGAN, GLIDE, Latent Diffusion, Stable Diffusion and ADM, first on uncompressed images and then after a random crop resized to 200 by 200 pixels and JPEG-compressed at a random quality between 65 and 100 (arXiv 2211.00680). The best detector, Grag2021, averaged 92.8 AUC and 75.8 percent accuracy uncompressed; after resizing and compression it averaged 90.8 AUC and 61.5 percent accuracy. The Wang et al. classifier averaged 78.6 AUC and 59.5 percent accuracy uncompressed, 75.0 and 55.8 percent compressed.

The authors' summary line is that "the performance is acceptable in terms of AUC, but almost random in terms of accuracy". The two metrics diverge because AUC asks whether the detector ranks fakes above reals at some threshold, while accuracy uses the threshold fixed during training, which no longer fits images of a different origin. Their Table 2 shows the fix they tried: a Grag2021 model trained on Latent Diffusion averaged 66.9 percent accuracy and 89.2 AUC; fused with the ProGAN model and calibrated, it reached 87.8 percent accuracy at 94.1 AUC.

06Why does each table hold only for its generator list?

Every number above is conditional on a generator list and a damage protocol. The 92.6 mean average precision of 2020 is for 11 GAN-era generators on clean images. The 72.1 percent of 2023 is for eight generators in GenImage, seven of them diffusion models, on clean images. The 61.5 percent of Corvi et al. is for 12 generators after resizing and JPEG. On Deepfake-Eval-2024, where the generator of each of 1,975 circulated images is unknown, three open-source image detectors scored 0.56, 0.52 and 0.53 AUC against 0.94, 0.99 and 0.98 on their original benchmarks, an average fall of 45 percent (arXiv 2503.02857, Table 3).

No single benchmark score predicts how a detector performs on a generator released after the benchmark was built. Face recognition went through the same cycle: the LFW dataset saturated at an accuracy of 99.83 percent and stopped separating systems, and GenImage's 99.9 percent same-generator cells are the equivalent ceiling. The number that carries information is the off-diagonal one.