Deepfake detection models ranked: 15 detectors by cross-dataset AUC

This page belongs to the deepfake detection hub and is the ranking view of the DeepfakeBench results. The frame-level AUC table on this site carries the same cells with every test set; this page orders them.

01How were the deepfake detection models ranked?

Every number here comes from Table 3 of the DeepfakeBench paper (arXiv 2307.01426). The authors trained 15 published detectors under one codebase on FaceForensics++ at the c23 compression level, then tested each on 14 sets: six within-domain splits of FaceForensics++ (c23, c40 and the four single-method subsets) and eight datasets the models never saw, which are Celeb-DF v1, Celeb-DF v2, DeeperForensics-1.0, the Google and Jigsaw DFD set, DFDC, the DFDC Preview, FaceShifter and UADFV. The table caption states that "all detectors are trained on FF-c23 and evaluated on other data".

The metric is frame-level AUC. The paper writes that "our benchmark currently adopts the frame level evaluation", samples 32 frames per video, and reports "the average value of the top-3 metrics (e.g., average top-3 AUC)" for each cell. The ranking column on this page is the paper's own cross-domain "Avg." column, the mean of the eight unseen-set AUCs. The within-domain average is the mean of the six FaceForensics++ columns. Nothing on this page is recomputed; the only derived column is the best unseen set, which is the largest of the eight published cells in each row.

02All 15 detectors ranked by cross-dataset average

The paper groups detectors into three families: naive (a plain image classifier), spatial (a method built around a pixel-domain cue) and frequency (a method that reads the spectrum). The family labels below are the paper's.

Frame-level AUC, all detectors trained on FaceForensics++ c23, DeepfakeBench Table 3 (2023). Xception and EfficientNet-B4 share fifth place because the table prints the same cross-domain average, 0.7718, for both.

Three things stand out. The top nine detectors sit inside a band of 0.0230, from 0.7875 to 0.7645, so the ordering among them rests on the third decimal place. Every detector with a within-domain average above 0.93 lands between 0.6985 and 0.7875 across datasets, a loss of roughly 0.16 to 0.24. And the best unseen set is UADFV for eleven of the fifteen rows, which says more about UADFV, a set of 49 real and 49 fake videos, than about the detectors.

03Why does a plain Xception sit in the top five?

The DeepfakeBench authors raise this themselves. Xception and EfficientNet-B4 are naive detectors, meaning an image classifier trained on real and fake crops with no cue designed for forgeries, yet they tie for fifth on the cross-domain average and Xception has the second-highest within-domain average at 0.9450. The paper writes that naive detectors "register high AUC values that are comparable to more sophisticated algorithms" and suggests that "the performance gap could be a product of these additional factors rather than the intrinsic superiority of the method", the factors being pre-training, augmentation and training frames, which the benchmark holds constant.

The gap between first place and the plain classifier is 0.0157 on the cross-domain average. Eight of the fifteen detectors use an Xception backbone, according to the table's backbone column, so much of the ranking is a comparison of what is fed to Xception and how it is trained, not of fundamentally different networks.

04Which detector wins on which unseen set?

The overall leader is not the leader everywhere. UCF has the most top-three finishes across the eight unseen sets, five, against three for SPSL, and UCF also has the highest within-domain average at 0.9527. SPSL wins both Celeb-DF columns, 0.8150 on v1 and 0.7650 on v2, and that is where its margin comes from. FWA, twelfth overall, scores 0.9334 on DeeperForensics-1.0, the highest cell in that column, because the warping artefact it looks for is prominent in that set's fakes. Face X-ray, fourteenth overall, leads on FaceShifter at 0.6553 and collapses to 0.5531 on DeeperForensics-1.0.

The cross-dataset drop by detector page lists the Celeb-DF v2 and DFDC cells for every row and the size of each fall, and explains why a cue that wins on one generator loses on the next.

05Why does the ranking hold only under these conditions?

The ranking holds for FaceForensics++ c23 training and frame-level AUC only; change the training set, the compression, the metric or the test sets and the order changes. Each condition matters on its own. Training on c23 means the models learned from videos compressed with H.264 at constant rate quantization parameter 23; a model trained on c40 or on raw frames would learn different traces. Frame-level AUC scores each sampled frame as a separate item, so a video with one obvious frame and 31 clean ones counts 32 times; video-level scoring, which the paper lists as future work, aggregates first. And the eight unseen sets are the datasets the benchmark shipped with in 2023, so any generator released since is absent by construction.

The benchmark's own limitation section says it will "further enhance the benchmark by incorporating video-level detectors and evaluation metrics". A reader who needs a ranking on circulated 2024 fakes should look at the Deepfake-Eval-2024 benchmark instead, where the best off-the-shelf open-source video model scored 0.63. The two rankings do not share a single detector or a single test set, so they cannot be merged.

06Why do numbers differ between sources?

The DeepfakeBench paper disagrees with itself in one place. Section 4.2 gives UCF's within-domain average as 95.37 percent; Table 3 prints 0.9527 for the same quantity. This page uses the table, because the table is the row the other fourteen averages come from, and the difference is 0.0010. The GitHub README reproduces a results table that it says is cited from the paper and adds that the v2 benchmark "will update the below table soon", so a reader comparing this page with the repository in a later year may find new numbers; this page is pinned to the 2023 paper. Where this site's results table and this page quote the same detector, the cell is identical.

07How to cite a rank from this page

A rank is a statement about one table, so quote it with its conditions: detector, training set and compression, metric, protocol and year. A usable sentence is "SPSL ranked first of 15 detectors by cross-dataset average frame-level AUC, 0.7875, when trained on FaceForensics++ c23 in DeepfakeBench (2023)". A sentence that drops the conditions, such as "SPSL is the best deepfake detector", is not supported by the table, and a sentence that converts the AUC into an accuracy is wrong, because AUC averages over every threshold while accuracy and the false positive rate depend on one.