Cross dataset deepfake detection: the drop by detector, model by model

This page belongs to the deepfake detection hub and is the per-model companion to the cross-dataset generalisation explainer. That page says why the drop happens; this one lists it detector by detector from one table, with the conditions the numbers were produced under.

01Where the numbers come from

All figures on this page come from Table 3 of the DeepfakeBench paper (arXiv 2307.01426), which trains 15 detectors on FaceForensics++ at the c23 compression level and tests each on six FaceForensics++ splits (c23, c40 and the four single-method subsets) and on eight datasets the models never saw: Celeb-DF v1 and v2, DeeperForensics-1.0, the Google and Jigsaw DFD set, DFDC, the DFDC Preview, FaceShifter and UADFV. The metric is frame-level AUC, computed over 32 frames taken from each video; the paper writes that "our benchmark currently adopts the frame level evaluation to build a fair basis for comparison among detectors".

Two averages matter here. The within-dataset average is the mean AUC over the six FaceForensics++ test columns, and the cross-dataset average is the mean over the eight unseen sets. The published results table on this site carries the same rows; this page adds the size of each drop. The drops are subtractions of two published cells and are labelled as such, so a reader can recompute them from the table.

02How far does each detector fall?

The table lists every detector in the paper's Table 3, grouped as the paper groups them: naive (a plain classifier), spatial (a method that looks for a pixel-domain artefact) and frequency (a method that looks at the spectrum). The last two columns are computed from the cells to their left.

Frame-level AUC, all detectors trained on FaceForensics++ c23, from DeepfakeBench Table 3 (2023). A plus sign marks a model whose unseen-set score is higher than its in-dataset score.

Three readings follow from the table. First, every detector that learned the in-dataset task well, meaning a FaceForensics++ c23 AUC above 0.95, lands between 0.6955 and 0.7650 on Celeb-DF v2 and DFDC, a band only 0.07 wide. Second, the largest single drop belongs to Face X-ray, which falls from 0.9592 to 0.6786 on Celeb-DF v2 and to 0.5531 on DeeperForensics-1.0, because its blending-boundary cue depends on how a fake was composited. Third, the one exception to the rule, Meso4, is not a success: it scores 0.6077 in-dataset, so there is little learned behaviour to lose.

03Which detector keeps the most on unseen sets?

SPSL, a frequency method, has the highest cross-dataset average at 0.7875 and the highest Celeb-DF v2 score at 0.7650, while UCF has the highest in-dataset average at 0.9527 and the highest cross-dataset count of top-three finishes. The two rankings do not agree, which is the central point of running both protocols. The DeepfakeBench authors draw the same conclusion in prose: naive detectors such as Xception and EfficientNet-B4 "register high AUC values that are comparable to more sophisticated algorithms", and the gap between the best purpose-built method and a plain classifier is 0.0157 on the cross-dataset average.

Per dataset, the leaders change again. FWA scores 0.9334 on DeeperForensics-1.0, where most others sit between 0.80 and 0.88; Face X-ray leads on FaceShifter at 0.6553; UCF leads on UADFV at 0.9528. No detector in the table leads on more than two of the eight unseen sets.

04Does the drop depend on the test set or the detector?

Mostly on the test set. Reading the table by column, DFDC is the hardest unseen set for every well-trained model, with scores from 0.6955 to 0.7191; Celeb-DF v2 is next, from 0.7319 to 0.7650; UADFV is the easiest, above 0.93 for all but the naive MesoNet variants. The spread across detectors within a column is far smaller than the spread across columns for one detector, so the choice of unseen set moves a published score more than the choice of architecture does. The DeepfakeBench paper describes how the sets were preprocessed identically so that the columns are comparable.

The same effect appears inside FaceForensics++. In the paper's cross-manipulation experiment, where a detector trains on one of the four methods and tests on the others, "their performance significantly deteriorates when faced with FF-FS forgeries" and the NeuralTextures split "poses challenges for almost all detectors". A detector can therefore fail on a new method even when the source videos, the compression and the identities are unchanged.

05What the drop does not tell you

A cross-dataset AUC is still a laboratory number. Celeb-DF v2 and DFDC are curated sets with known generators and controlled compression; a clip from a social platform has neither property, and the in-the-wild benchmarks report a second drop on top of the one tabulated here. The 2023 table also predates diffusion-based swaps and the 40-method DF40 benchmark, so a 0.77 cross-dataset average describes 2020-era fakes.

A frame-level AUC is not an accuracy either. It averages over every threshold, and a deployment picks one. The metrics page on this site covers the translation from AUC to a working operating point. The quotable caveat for this page is the one the whole table supports: no single benchmark score predicts how a detector performs on manipulations it has not seen.

06How to cite a number from this table

Give the detector, the training set and compression (FaceForensics++ c23), the test set, the metric (frame-level AUC), the benchmark version (DeepfakeBench, 2023) and the table (Table 3). "Xception scores 0.7365 frame-level AUC on Celeb-DF v2 when trained on FaceForensics++ c23 (DeepfakeBench Table 3, 2023)" is a complete citation; "Xception is 74 percent accurate on Celeb-DF" is not, because it drops the metric, the protocol and the date. The same conditions apply to the human or AI studies, the deepfake voice benchmarks and the liveness detection tests on this site: every number carries its test set and its date.