AI image detection: how a detector scores an image, and what it means

This explainer belongs to the deepfake detection hub. The scores it quotes come from the benchmarks compared on the generated-image detection benchmarks page, which lists each one's generator set.

01What does an AI image detector actually learn?

An AI image detector is a binary classifier: it takes an image and returns one number, usually between 0 and 1, that is higher when the image looks like the generated images it was trained on. It is trained on a set of real photographs and a set of generated images and adjusts its weights until the two separate. Nothing in that process tells it what a face, a street or a cat is. What separates the two sets, in practice, is low-level texture: the way a generator's final upsampling layers build pixels, which differs from the way a camera sensor and a JPEG encoder build them.

Frank et al. (ICML 2020) showed this directly. Transformed into frequency space, GAN images "exhibit severe artifacts that can be easily identified", and the artefacts are "caused by upsampling operations found in all current GAN architectures" (arXiv 2003.08685). A detector does not need to be shown that pattern; a plain image classifier finds it on its own, which is why the standard backbones used for object recognition, ResNet and its successors, are also the standard detectors.

02Why did one classifier generalise across eleven generators in 2020?

Wang, Wang, Zhang, Owens and Efros trained a ResNet-50 on a set of 720,000 training images, real photographs and ProGAN outputs in equal numbers across 20 LSUN categories, with ProGAN as the only generator, and tested it on 11 CNN generators including StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN and a DeepFakes face set (arXiv 1912.11035). With blur and JPEG compression applied to 10 percent of training images, the mean average precision across the 11 was 92.6, where chance is 50: 99.6 on StyleGAN, 84.5 on BigGAN, 89.0 on the DeepFakes set. The abstract's reading was that a classifier trained on "only one specific CNN generator (ProGAN) is able to generalize surprisingly well to unseen architectures".

The title of that paper ends "for now", and the authors meant it. The shared flaw was a property of a generation of GANs built from similar convolutional upsampling. A detector that learns a family trait generalises within the family and no further.

03What changed when diffusion models arrived?

Diffusion models build an image by a different route and leave a different trace. Corvi et al. (2022) took four detectors trained on ProGAN only and tested them on 12 generators including Latent Diffusion, Stable Diffusion, GLIDE and ADM (arXiv 2211.00680). The best of them averaged 92.8 AUC on clean images but only 75.8 percent accuracy; the Wang et al. classifier averaged 78.6 AUC and 59.5 percent accuracy, and on Stable Diffusion images specifically it scored 65.9 AUC with 50.9 percent accuracy, which is chance.

GenImage (2023) measured the same thing at scale, with 1,350,000 images from eight generators, seven of them diffusion models, paired with 1,331,167 ImageNet photographs (arXiv 2306.08571). A ResNet-50 trained on one generator scored above 98.5 percent accuracy on that generator and between 58.6 and 72.1 percent on average across all eight. The Wang et al. classifier, run as a baseline, averaged 64.2 percent when trained on Stable Diffusion V1.4. The lesson of 2020 inverted: the detector was tied to its generator again, because the new generators no longer shared the old flaw.

04How far do scores fall on images found online?

Deepfake-Eval-2024 collected 1,975 images that social media and detection-platform users flagged in 2024, from 88 websites, with no record of which generator made each one (arXiv 2503.02857). Three open-source image detectors were run off the shelf. UFD scored 0.56 AUC against 0.94 on its original benchmark; DistilDIRE 0.52 against 0.99; NPR 0.53 against 0.98 (Table 3). Averaged, image detectors lost 45 percent of their AUC, and the paper notes that many off-the-shelf models sit "close to 0.5, the same as random guessing".

Fine-tuning on 60 percent of the set and testing on the other 40 percent lifted NPR to 0.73 AUC but left UFD and DistilDIRE at 0.56 (Table 4); the average gain for images was 4.5 percent, against 57.6 percent for video and 80.6 percent for audio. The best anonymised commercial image model reached 0.90 AUC and 0.82 accuracy (Table 5), and no commercial model of any kind reached 90 percent accuracy. The same paper's video and audio detectors fell by 50 and 48 percent.

05What a score from an image detector means

A detector's output is a position on a scale, not a verdict. Three things turn it into a probability, and a user usually has none of them.

First, the threshold. The detector calls an image fake above some cut-off chosen during training. Corvi et al. found that the same ProGAN-trained model kept 90.8 AUC after images were resized and JPEG-compressed, while its accuracy fell to 61.5 percent, because the training threshold no longer matched the compressed images; in their words, performance was "acceptable in terms of AUC, but almost random in terms of accuracy". A score of 0.8 from a detector whose threshold has drifted means less than it looks.

Second, the generator list. Every published score is conditional on the generators in the test set. A detector reported at 0.98 AUC was reported on known generators; on 2024 images from unknown ones the same detector scored 0.53. Unless the image's generator is in the training set's family, the published score does not apply.

Third, the base rate. Deepfake-Eval-2024's authors posit that a useful detector should reach at least 90 percent accuracy on their set, and none they tested did. At accuracies near 60 percent, a "fake" verdict on an image that was probably real to begin with is more likely to be wrong than right. This is why the checking a face video or photo yourself page puts the detector score last, after where the image came from and what else corroborates it, and why the provenance versus detection page treats a signed origin record as stronger evidence than any classifier output.

06Why do the same models score differently in different papers?

Numbers for the same detector differ between sources because the test sets, metrics and image handling differ. The Wang et al. classifier is 92.6 mean average precision in its own paper (11 GAN generators, clean images), 64.2 percent accuracy in GenImage (eight generators, trained on Stable Diffusion V1.4) and 59.5 percent accuracy with 78.6 AUC in Corvi et al. (12 generators, trained on ProGAN, clean images). None of these is wrong; each measures a different question. The glossary of deepfake detection terms separates average precision, AUC and accuracy, the three scales these figures use.

Face recognition offers the precedent. The LFW dataset saturated once deep networks reached an accuracy of 99.83 percent on its 6,000 pairs, and the field moved to harder sets. GenImage's same-generator cells at 99.9 percent are the same kind of ceiling: a benchmark that every system passes has stopped measuring anything, and the informative number is the one on unseen generators.