Deepfake detection benchmarks compared: FF++, DFDC, DeepfakeBench, DF40

This page is the comparison node of the deepfake detection hub. Each benchmark has its own page; the cross-dataset AUC table by detector and test set holds the full DeepfakeBench and DF40 numbers, and this page explains why those numbers cannot be read across benchmarks.

01The five benchmarks in one table

Every row's conditions are in its own paper, linked under Sources. The public dataset counts and licences table gives the sizes and access terms for the sets the first four rows draw on.

02What does each benchmark actually test?

FaceForensics++ tests whether a detector can recognise four named manipulations it was trained on, at a known compression. Its paper reports XceptionNet trained on all four methods at 99.26 percent accuracy on raw video, 95.73 on c23 and 81.00 on c40, and a 204-person user study at 68.69, 66.57 and 58.73 percent on the same material (arXiv 1901.08971). It is an in-domain test with compression as the only variable.

DFDC tests a competition: entrants trained on 119,154 released clips and were ranked by log loss on a private set of 10,000 clips, half of them organic internet content. The winning entry scored 0.4279, and the paper reports that a model trained only on DFDC kept an AP of 0.753 and ROC-AUC of 0.734 on real in-the-wild deepfakes the organisers collected (arXiv 2006.07397). It measures what the best teams of 2020 could do, not what any published detector does.

DeepfakeBench and DF40 test generalisation under a fixed protocol. DeepfakeBench trains 15 re-implemented detectors on FF++ c23 and scores them on 14 test sets with 32 frames per video; its Table 3 shows within-domain averages of 0.94 to 0.95 for the stronger detectors and cross-domain averages of 0.66 to 0.79 across all 15 (arXiv 2307.01426). DF40 keeps the same preprocessing and training code but changes the question: 40 generators split into face swapping, face reenactment, entire face synthesis and face editing, with protocols that hold out a forgery family, a data domain, or both (arXiv 2406.13495).

Deepfake-Eval-2024 tests the world: media collected in 2024 from social platforms and users of a detection service, in 52 languages. Open-source detectors' AUC fell by 50 percent for video, 48 percent for audio and 45 percent for images against their original benchmarks (arXiv 2503.02857).

03Why are the scores not comparable?

The metrics differ in kind. Accuracy depends on a threshold; log loss punishes confident errors; AUC averages over all thresholds and ignores calibration. A detector can hold a high AUC and a poor accuracy at the same time if its scores are shifted, and DeepfakeBench's appendix shows the Xception baseline moving from an average AUC of 0.6250 to 0.8160 on the same test sets once augmentation and pre-training are added, a change of recipe rather than of architecture.

The training sets differ. DeepfakeBench fixes FF++ c23 for every detector; DF40 trains on one forgery family within one domain; the DFDC winner trained on DFDC; Deepfake-Eval-2024 takes each model as its authors released it. The test material differs again: actors under studio light, celebrities on YouTube, found clips re-encoded by platforms. And the unit differs: DeepfakeBench scores frames and states that it "currently adopts the frame level evaluation", DF40 reports video-level AUC only for its one video model, DFDC scored clips.

Even the two benchmarks that share code diverge in a detail that moves numbers. DF40 states that "unlike DeepfakeBench, which does not involve using the validation set, we use the validation set" to pick checkpoints. The same architecture, the same preprocessing and a different stopping rule give a different table.

04Which benchmark answers which question?

Whether a detector learned the training manipulations: FaceForensics++, or the within-domain columns of DeepfakeBench. Whether it survives compression: the FF++ c23 and c40 pair, where XceptionNet drops 18 points of accuracy and people drop 10. Whether it transfers to another curated set of the same era: the cross-domain columns of DeepfakeBench, where the best cross average is 0.7875. Whether it transfers to a generator family it never saw: DF40 Protocol 1, where Xception trained on face swaps scores 0.810 on entire face synthesis and the reverse direction scores about 0.6. Whether it works on last year's internet: Deepfake-Eval-2024.

What a competition leaderboard answers is narrower than it looks. DFDC's winning log loss of 0.4279 came from one team's ensemble on one private set; the paper reports that the same winner scored 0.1983 on the DFDC-only portion and 0.6605 on the real-only portion, so one number already hides two regimes.

05What one detector scores across them

The figure below follows a single architecture, Xception, through the protocols that share a metric. Under DeepfakeBench it scores 0.9637 within domain on FF++ c23, 0.8261 at c40, 0.7365 on Celeb-DF v2 and 0.7077 on DFDC. Under DF40, trained on the face-swapping methods in the FF++ domain, it scores 0.991 on those swaps, 0.810 on entire face synthesis in the same domain, 0.546 averaged over the unknown-domain sets and 0.394 on HeyGen videos, below chance. The DF40 authors' second finding states the consequence for the field: "Existing well-designed (SoTA) detectors may not have obvious advantages over the baseline".

06What none of them can show

No benchmark above scores an injected stream in a live identity check; the injection attack test gap page explains why the artefact there is a software path rather than a file. None reports how a person would do on the same material, which the human and machine accuracy on DFDC videos page covers from a separate study. And the four face benchmarks say nothing about voice: the audio generalisation test on found recordings is the equivalent of Deepfake-Eval-2024 for speech, with its own metric.

No single benchmark score predicts how a detector performs on unseen manipulations. A quoted figure needs its benchmark, its training set, its metric and its date before it means anything, and the table at the top of this page is the checklist.