AUC score explained: what the area under the ROC curve measures

This page is the definition entry for the metric that most pages in the deepfake detection hub quote. The broader metrics page covers equal error rate, accuracy and false positive rate beside it; this one stays on the AUC and what its value does and does not tell you.

01What does an AUC score measure?

A detector does not output "real" or "fake". It outputs a score, and a threshold turns the score into a decision. Sweep the threshold from strict to lenient and record, at each setting, the share of fakes caught (the true positive rate) and the share of real samples wrongly flagged (the false positive rate). Plotting one against the other draws the ROC curve: in Fawcett's description, "ROC graphs are two-dimensional graphs in which tp rate is plotted on the Y axis and fp rate is plotted on the X axis" (Fawcett, 2006).

The AUC is the area under that curve. Because the curve lives in the unit square, "its value will always be between 0 and 1.0". A classifier that guesses at random traces the diagonal from (0, 0) to (1, 1), which encloses an area of 0.5, so Fawcett adds that "no realistic classifier should have an AUC less than 0.5". A detector that gives every fake a higher score than every real sample traces the top left corner and scores exactly 1.0.

02Why is AUC a probability?

The number has a second reading that is often more useful than the geometry. Fawcett states that the AUC of a classifier is "equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance", and notes that this is the same quantity as the Wilcoxon rank statistic (Hanley and McNeil, 1982). For a deepfake detector the sentence reads: pick one fake frame and one real frame at random from the test set; the AUC is the chance the detector gives the fake the higher score.

That reading makes the reference points concrete. At 0.5 the detector orders a random fake above a random real half the time, which is a coin toss. At 0.9637, the frame-level AUC of the Xception baseline on FaceForensics++ c23 in the DeepfakeBench paper (arXiv 2307.01426), it orders them correctly about 96 times in 100. At 0.7365, the same model's AUC on Celeb-DF v2, it does so about 74 times in 100. The same paper's Table 3 gives these and every other row that the cross-dataset scores by model table on this site carries.

03How does AUC differ from accuracy?

Accuracy is the share of samples classified correctly at one threshold. It moves with the threshold and with the class balance of the test set. Fawcett's paper opens with the machine learning community's "realization that simple classification accuracy is often a poor metric for measuring performance", and section 4.2 gives the property that fixes it: "ROC curves have an attractive property: they are insensitive to changes in class distribution". The AUC inherits that property. A test set with four fakes for every real, like the FaceForensics++ release, does not inflate it the way it inflates accuracy.

The two numbers can therefore disagree. The WildDeepfake paper reports accuracy, not AUC, and shows baseline image detectors scoring above 95 percent on the FaceForensics++ Deepfake HQ split and below 70 percent on internet fakes (arXiv 2101.01456); the WildDeepfake dataset page carries the table. An accuracy is only comparable with another accuracy at the same threshold on a set with the same real to fake ratio. An AUC is comparable with another AUC on the same test set, which is why the results tables on this site lead with AUC wherever the source reports it.

04How does DeepfakeBench report AUC?

DeepfakeBench's evaluation module uses four metrics: "accuracy (ACC), the area under the ROC curve (AUC), average precision (AP), and equal error rate (EER)". The paper notes that some detection papers compute these per frame and others per video, "leading to unfair comparisons", and states: "Our benchmark currently adopts the frame level evaluation to build a fair basis for comparison among detectors". Each video contributes 32 sampled frames, every frame is scored on its own, and the AUC is computed over all frames in the test set. The value reported is the average of the top three AUCs recorded during training, with every detector trained on FaceForensics++ c23 and tested elsewhere without retraining.

The frame-level choice matters when you compare numbers across papers. A video-level AUC combines the frame scores of a clip into one score before ranking, so it is a different quantity measured on a different number of samples. The DeepfakeBench authors list video-level detectors and metrics as future work. A frame-level AUC and a video-level AUC for the same model on the same set are two different numbers, and this site labels each row with which it is.

05What does a high AUC not tell you?

Three things. It does not tell you the error rates at the threshold you would actually use: a detector with AUC 0.96 can still be run at a threshold that flags one real video in five. Fawcett is explicit that "it is possible for a high-AUC classifier to perform worse in a specific region of ROC space than a low-AUC classifier", so two curves can cross and the one with the smaller area can be the better choice at a low false positive rate. The scikit-learn implementation exposes this through its max_fpr argument, which returns a standardised partial AUC over the low false positive region only (scikit-learn, roc_auc_score).

It does not transfer to another test set. The DeepfakeBench Xception row runs from 0.9637 on FaceForensics++ c23 to 0.7077 on DFDC with the same weights, and the fakes in both sets are made by the kinds of pipeline described on the how deepfakes are made page; a newer generator is a new distribution again. And it does not say how the ordering was achieved. A detector that ranks fakes above reals by learning the compression signature of one dataset has a high AUC on that dataset and no claim on the next one. No single benchmark score predicts how a detector performs on unseen manipulations.

06How is AUC computed in practice?

Most published deepfake results use roc_auc_score from scikit-learn or an equivalent. The function takes the true labels and either probability estimates or "non-thresholded decision values", sorts the scores, and integrates the resulting ROC curve; the library documentation describes it as computing the "Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores". No threshold is chosen at any point, which is the mechanical reason the AUC is threshold free.

Two related numbers appear in older papers. The Gini coefficient is a rescaling of the same quantity, twice the AUC minus one in the scikit-learn note, so that random guessing scores 0 and perfect ranking scores 1; Fawcett gives the same relation as "Gini + 1 = 2 · AUC". The Wilcoxon rank statistic that Fawcett cites is the AUC computed directly from pairwise comparisons of positive and negative scores rather than from the curve. All three describe one ordering. When a paper quotes any of them, the question to ask is the same as for accuracy: which test set, which split, and frame or video. The definition of a deepfake page shows how the answer changes the headline number for the same detector.