AUC, EER, accuracy and false positive rate: how deepfake detectors are scored

This page is the metrics companion to the published detector results on this site, and it is the part of the deepfake detection benchmark hub most readers need first. Every figure in a detection paper is one of a small set of quantities, and the same quantities appear, under older names, in face verification tables.

01What an AUC score means

A detector does not say "fake". It produces a number, usually between 0 and 1, that is meant to be higher for manipulated faces than for real ones. To get a decision, someone picks a threshold: above it, fake; below it, real. The ROC curve (receiver operating characteristic) is what you get when you slide that threshold from one end to the other and, at each position, plot the true positive rate (fakes correctly flagged) against the false positive rate (real media wrongly flagged).

The AUC, or area under the ROC curve, summarises that whole curve in one number. An AUC of 1.0 means some threshold separates every fake from every real sample. An AUC of 0.5 is what a coin toss would score. The intuitive reading is this: pick one random fake and one random real sample; the AUC is the probability that the detector gives the fake the higher score.

Two things follow. First, AUC is threshold-free. It tells you how good the ordering is, not how good any particular decision rule is. Second, AUC is insensitive to how many fakes and reals are in the test set, which is why benchmark papers prefer it over accuracy. DeepfakeBench, the shared evaluation codebase from the Chinese University of Hong Kong, Shenzhen, reports frame-level and video-level AUC as its primary metrics, alongside accuracy, EER, precision-recall and average precision (DeepfakeBench repository).

02The ROC curve and ROC AUC

Reading the curve

The curve starts at the bottom left (threshold so high that nothing is flagged: no false positives, no true positives) and ends at the top right (threshold so low that everything is flagged). A detector that is better than chance bows up toward the top left corner. The diagonal is chance. The further the curve sits from the diagonal, the larger the area beneath it.

Where on the curve a system operates is a separate choice from how good the curve is. A detector with AUC 0.96 can still be run at a threshold where it flags one in five real videos, if whoever deployed it wanted to catch almost every fake. The curve tells you the trade-off that is available; the threshold is the trade-off that was taken.

Why AUC can be high while accuracy is poor

Accuracy is the share of all samples classified correctly at one threshold. It depends on the threshold and on the class balance of the test set. On a set with 1,000 real videos and 4,000 fakes, a rule that calls everything fake scores 80% accuracy and 0.5 AUC. On a set that is 1:1 the same rule scores 50%. This is why two papers can quote very different accuracies for similar detectors, and why the results on this site are AUC wherever the source reports it.

The cross-dataset numbers in the DeepfakeBench results table show the other direction: a detector trained on FaceForensics++ at c23 compression with an in-dataset frame-level AUC of 0.9637 (Xception) drops to 0.7365 on Celeb-DF v2 and 0.7077 on DFDC when tested without retraining (DeepfakeBench results table, from the NeurIPS 2023 paper). The ordering is still better than chance, but a threshold tuned on the first set would be badly placed on the others. The page on cross-dataset generalisation walks through why.

03False positive rate and the operating point

The false positive rate (FPR) is the share of real samples that the detector flags as fake at the chosen threshold. Its mirror is the false negative rate: fakes that pass as real. In biometrics the same pair is called false match rate and false non-match rate for 1:1 verification, and false positive and false negative identification rate for 1:N search; NIST's demographic effects report defines them exactly that way (NISTIR 8280).

1% FPR as the practical bar

For anything that acts on a verdict, the false positive rate is the number that matters, because every false positive is a real person's real video treated as a forgery. Reporting the true positive rate at a fixed, low false positive rate (for example, TPR at 1% FPR) is how face verification has been scored for years, as TAR at FAR. Deepfake detection papers report it less often than AUC, which is a gap worth noticing when you read one: a detector with a strong AUC may still have a weak true positive rate once you pin the false positive rate to 1%, because much of its AUC can come from the part of the curve where false positives are high.

When this site's results table carries a figure at a fixed operating point, the row says so. When a source only reports AUC, the row says AUC and nothing else is inferred from it.

04Equal error rate

The equal error rate (EER) is the point on the curve where the false positive rate equals the false negative rate. It collapses the trade-off to one number that, unlike AUC, is tied to a specific threshold. Lower is better. An EER of 10% means that at the balancing threshold, one in ten reals is flagged and one in ten fakes is missed.

EER is the usual summary in speaker verification and in the ASVspoof voice spoofing challenges, and it appears in deepfake detection papers that borrow from that tradition. It is a fair way to compare systems when nobody has agreed on an operating point, and a poor guide to deployment, because almost no real system wants its false positives and false negatives to be equal. The voice deepfake benchmarks page covers how the ASVspoof challenges report their results.

05Accuracy, precision, recall and when they mislead

Accuracy is correct decisions divided by all decisions. Precision is the share of flagged items that really are fake. Recall is the share of fakes that were flagged (the same quantity as the true positive rate). All three depend on the threshold and on the mix of real and fake in the test set, so they travel badly between papers.

They are not useless. Precision answers the question a moderator asks ("of the videos this tool flagged, how many were actually fake?"), and recall answers the one a platform asks ("how many fakes got through?"). They are the right numbers for a specific deployment with a specific threshold and a known base rate. They are the wrong numbers for ranking methods across papers, which is why this site's results table leads with AUC and labels any accuracy figure with the set and threshold it came from.

A quick check when you read a paper: if a figure is called "accuracy" and the test set is heavily unbalanced, ask what a constant guess would have scored. If the paper does not say how many real and fake samples the test set holds, the accuracy figure cannot be interpreted at all.

06The face verification cousins: TAR at FAR

Face recognition solved the reporting problem earlier and more strictly. A 1:1 verification system is scored by its true accept rate (TAR) at a fixed false accept rate (FAR) set by the benchmark, and NIST's evaluations publish false non-match rates at fixed false match rates. The WebFace260M paper that once lived on this domain reported its training set's effect in exactly those terms, a relative 40% reduction in failure rate on IJB-C (arXiv 2103.04098).

The mapping is direct: TAR is the true positive rate, FAR is the false positive rate, and "TAR at FAR" is one point on the ROC curve. The face verification metrics page explains that table format, and the discipline it encodes, reporting the operating point every time, is the one this site asks of deepfake detection results.

07Metrics in the results table

Every row of the published detector results names its metric, its train set, its test set, the compression or protocol and the date checked, so the number can be compared with another row only when those match. The how to read a detection paper page shows where to find each of those in a paper, and the evaluate a detector protocol shows how to produce them for your own data.