Deepfake detection accuracy vs AUC: why a 99 percent score can mislead

This page sits in the metrics part of the deepfake detection hub. The scoring metrics overview defines accuracy, AUC, EER and false positive rate side by side; this page is about the single most common misreading, an accuracy figure quoted without its conditions.

01What does accuracy measure, and what does it need?

Accuracy is the number of correct labels divided by the number of items. A detector produces a score per frame or clip, a threshold turns the score into "real" or "fake", and accuracy counts how often that label matched the ground truth. Two choices sit inside the number that are not inside the detector: the threshold, which the authors picked, and the mix of real and fake items in the test set, which the dataset's builders picked.

Both choices move the figure. In FaceForensics++ the test split is 140 videos drawn from a set with 1,000 pristine and 4,000 manipulated videos, so a label of "fake" for every frame would already be right most of the time (arXiv 1901.08971). Fawcett's introduction to ROC analysis (Pattern Recognition Letters, 2006) opens from "the realization that simple classification accuracy is often a poor metric for measuring performance", and names class skew as the reason (Fawcett 2006).

02How can a 99 percent detector be useless?

The FaceForensics++ paper supplies the cleanest case, because it tests the same detectors on the same fakes at three compression levels. XceptionNet, trained on all four manipulation methods, reaches an accuracy of 99.26 percent on raw video, 95.73 percent on the high quality release and 81.00 percent on the low quality release. Nothing about the fakes changed except the H.264 quantisation parameter, 23 for high quality and 40 for low. The other baselines fall further: the steganalysis features with an SVM drop from 97.63 to 55.98 percent, and Cozzolino et al.'s method from 98.57 to 58.69 percent, both close to the 50 percent a coin would score on a balanced set.

Binary detection accuracy in percent on the FaceForensics++ test split of 140 videos. Source: Rössler et al., ICCV 2019, Table 1, read 9 November 2026.

A reader who saw only the raw column would rank five of these six detectors within three points of each other. The low quality column spreads them across 25 points. The 99 percent figure was true, and said almost nothing about social media video, which the paper notes is compressed with the same codec.

03What does AUC change?

AUC is the area under the ROC curve, the curve traced by true positive rate against false positive rate as the threshold sweeps from one extreme to the other. Because every threshold is included, no single operating point is chosen for the detector, and Fawcett shows that the curve is "insensitive to changes in class distribution", so a test set with ten times more fakes than real items draws the same curve. The value is "equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance", with 0.5 for random ranking and 1.0 for perfect separation. The AUC score page covers the construction in full.

That is why the DeepfakeBench authors chose frame-level AUC as the headline for 15 detectors across 14 test sets, computing accuracy, average precision and EER alongside it (arXiv 2307.01426). AUC lets a detector trained on FaceForensics++ be compared on Celeb-DF, where the real and fake counts differ, without the class balance leaking into the score. The DeepfakeBench results by test set are all AUC for this reason.

04When do accuracy and AUC disagree?

Deepfake-Eval-2024 (2025) prints both for the same models on the same set, which makes the disagreement visible. Its Table S8 gives the off-the-shelf open-source results on fakes that circulated in 2024: the image detector UFD scores an AUC of 0.56, near chance, with an accuracy of 0.63, because its recall is 0.999 and its false positive rate 0.99; it calls almost every item fake, and most items in the set are fake, so accuracy rewards it. The audio detector P3 has the opposite profile, an AUC of 0.58 and an accuracy of 0.36, with recall 1.00 and a false positive rate of 1.00 (arXiv 2503.02857, Table S8). Both detectors rank items barely better than random; the accuracy numbers say two different things about them because of where the threshold sat and how the set was composed.

The converse also holds, and Fawcett states it: "It is possible for a high-AUC classifier to perform worse in a specific region of ROC space than a low-AUC classifier". A deployment that must keep false alarms at one in a thousand lives in one region of the curve, and the whole-curve average can hide what happens there. Face recognition has long reported results that way, as an error rate at a fixed operating point, and the face recognition accuracy page explains why.

05Which number should a reader trust?

Neither, on its own. The reading order set out in how to read a detection paper applies: find the training set, the test set and its split, the metric with its level and operating point, and the protocol, including compression and whether the test set was seen in training. An accuracy figure without a threshold and a class balance is not a result; an AUC figure without the test set is not a result either.

Three habits follow. Prefer AUC when comparing detectors across sets with different class balances, because the class balance cannot move it. Ask for the false positive rate at the operating point when a detector is to be used on real media, because that is the cost a deployment pays. And discount any figure measured on the fakes the detector was trained on: in DeepfakeBench, Xception scores 0.9637 AUC on FaceForensics++ c23 and 0.7365 on Celeb-DF v2 it never saw, and that gap, not the 0.96, predicts behaviour on new material. The three published accuracy ranges for seen, unseen and wild fakes put numbers on all three conditions. No statute does the work for the reader either: the deepfake laws in force set disclosure and removal duties and no accuracy requirement for detectors.

No single benchmark score predicts how a detector performs on unseen manipulations.