ROC AUC: how the curve is drawn and why detectors report it

This page is the drawing and reading companion to the AUC score explained page in the deepfake detection hub. That page defines the area and its 0.5 and 1.0 reference points; this one stays on the curve itself: how it is built, where the useful points on it are, and why detection benchmarks moved from accuracy to this curve.

01How is an ROC curve drawn?

A deepfake detector outputs a score for each input, higher meaning more likely fake, and a curve needs every one of those scores. Fawcett's procedure is a single sort followed by a scan: "we can simply sort the test instances decreasing by f scores and move down the list, processing one instance at a time and updating TP and FP" (Fawcett, 2006). Before any instance is processed the threshold is effectively infinite, nothing is called fake, and the curve starts at (0, 0). Each time the next instance in the sorted list is a fake, the true positive rate rises by one over the number of fakes; each time it is a real sample, the false positive rate rises by one over the number of reals. After the last instance everything is called fake and the curve ends at (1, 1).

The result is not a smooth line. "Any ROC curve generated from a finite set of instances is actually a step function, which approaches a true curve as the number of instances approaches infinity." In the paper's worked example of 20 instances, each step is labelled with the score threshold that produces it, and "lowering this threshold corresponds to moving from the 'conservative' to the 'liberal' areas of the graph".

In DeepfakeBench the instances are frames: "we sample 32 frames for each video for training and testing", every frame is scored on its own, and the curve is drawn over all frames in the test split with the fake class as the positive (Yan et al., 2023). A video-level curve would instead treat each clip as one instance with one combined score, which changes both the number of points and the shape.

02What do the points on the curve mean?

Each point is one possible detector, the one you would get by fixing the threshold at that score. Fawcett's description of the geography is the most useful guide to reading it. The point (0, 0) "represents the strategy of never issuing a positive classification"; (1, 1) is its opposite, calling everything fake; and "the point (0, 1) represents perfect classification". Between them, "one point in ROC space is better than another if it is to the northwest (tp rate is higher, fp rate is lower, or both)".

The two ends of a curve have names. Points near the left axis belong to detectors that are "conservative: they make positive classifications only with strong evidence so they make few false positive errors", at the cost of missing fakes. Points toward the upper right are "liberal: they make positive classifications with weak evidence", catching most fakes and flagging many real videos with them. Fawcett adds that because "many real world domains are dominated by large numbers of negative instances", the far left of the graph is usually the interesting part. For a detector screening a platform where almost every video is real, the operating region that matters is a narrow strip near zero false positive rate, and the metrics page treats a 1 percent false positive rate as the practical bar.

03Where is chance on the curve, and where is the equal error rate?

"The diagonal line y = x represents the strategy of randomly guessing a class." A detector that labels a random 70 percent of inputs as fake catches 70 percent of fakes and flags 70 percent of reals, which is the point (0.7, 0.7); vary the proportion and the point slides along the diagonal. A curve that hugs the diagonal carries no information, and "any classifier that appears in the lower right triangle performs worse than random guessing", which Fawcett notes can be fixed by negating its decisions. In DF40's one-versus-all heatmaps the authors colour 0.5 AUC white for exactly this reason: "white means 0.5 AUC (by chance)" (Yan et al., 2024).

The equal error rate is the one point on the curve where the two error rates match: the false positive rate equals the false negative rate, which is one minus the true positive rate. Geometrically it is where the curve crosses the line from (0, 1) to (1, 0), and it is reported as that shared error, so a lower number is better. DeepfakeBench records EER alongside AUC, AP and accuracy and uses it in its appendix to compare 13 detectors across datasets. EER is a single operating point chosen by a rule rather than by cost, which makes it comparable across papers but rarely the point anyone would deploy at; the cross-dataset generalisation page shows how the same models rank differently once the test set moves, whichever point is read.

04Why do benchmarks report AUC rather than accuracy?

Accuracy is the height of one point on the curve read at one threshold, and Fawcett's paper is partly an argument against relying on it. The "accuracy point" at a 0.5 threshold is, in his Figure 4, an operating point that is "suboptimal", and the best accuracy in his 20-instance example occurs at a threshold of about 0.54 rather than 0.5. Two detectors compared by accuracy are compared at whatever thresholds their authors happened to pick, which is why he recommends against "comparing model performance at a common threshold" when the scores are not calibrated.

The second reason is class balance. ROC curves "are insensitive to changes in class distribution", while accuracy moves with the ratio of fakes to reals, and Fawcett observes that class skews of ten to one and a hundred to one "are very common in real world applications". FaceForensics++ has four fakes for every real video, and its paper's Table 1 reports binary detection accuracy for its baselines: an accuracy of 99.26 percent for XceptionNet on raw video, 95.73 on the c23 release and 81.00 on c40 (Rössler et al., 2019). Those are honest numbers at one threshold on one class ratio. DeepfakeBench, four years later, states that "our benchmark currently adopts the frame level evaluation to build a fair basis for comparison among detectors" and reports AUC as the headline, so that detectors trained by different groups with different thresholds can be ranked on their ordering alone. The cross-dataset scores by model table on this site follows that choice and labels every row with its metric.

05How do two curves compare when they cross?

The area is one number, and two curves with the same area can have different shapes. Fawcett's section 6 gives the tool for choosing between them under a stated cost: an iso-performance line whose slope is set by the class and cost ratios, where "lines 'more northwest' (having a larger TP-intercept) are better". A detector is the best available for some cost and some class balance only if its point lies on the outer boundary of all the points, which the paper names "the ROC convex hull (ROCCH)".

For deepfake detection the practical version is simpler. Read the two curves in the strip that matches the deployment, usually the low false positive region, and prefer the one that is higher there even if its total area is lower. The AUC score explained page quotes Fawcett's warning that a high-AUC classifier can lose to a low-AUC one in a specific region; the curve is where that loss is visible and the area is where it is hidden.

06What should a reader check before trusting an ROC AUC?

Four things, each of which changes the curve without changing the detector. Whether the instances are frames or videos: DeepfakeBench and DF40 both report frame-level AUC as the primary metric, and DF40 reports video-level AUC only for its one video model. Which test set the curve was drawn on: the same Xception weights give 0.9637 on FaceForensics++ c23 and 0.7077 on DFDC in DeepfakeBench's Table 3. How the reported value was chosen: DeepfakeBench averages the top three AUCs recorded during training, while DF40 picks a checkpoint on a validation set, so the two are not the same statistic. And which class is positive: every page on this site treats the fake as the positive class, so a true positive is a caught fake and a false positive is a real video flagged.

The curve also says nothing about the cases on the documented deepfake incidents page, because no detector was run in any of them; it describes how a detector would behave on a test set of the same kind, not on the next clip. No single benchmark score predicts how a detector performs on unseen manipulations, and an ROC AUC is a single benchmark score drawn on one test set. The DF40 dataset page shows what happens to the same baseline when the forgery and the footage change: the area moves from 0.991 to 0.546.