How accurate are deepfake detectors? Seen fakes, unseen fakes, the wild

This page is the summary entry to the published results part of the deepfake detection hub. Every figure here is an AUC, and the accuracy versus AUC page explains why a benchmark reports that rather than a percentage; the full table the first two rows come from sits on the results page.

01What are the three conditions, in one table?

All DeepfakeBench figures are frame-level AUC with every detector trained on FaceForensics++ c23 and the reported value the average of the top three checkpoints (arXiv 2307.01426). Deepfake-Eval-2024 reports video-level AUC on a set of 45 hours of video, 56.5 hours of audio and 1,975 images (arXiv 2503.02857).

02How accurate are detectors on the fakes they trained on?

High. In DeepfakeBench Table 3, the within-domain average AUC across the six FaceForensics++ test sets is 0.9527 for UCF, 0.9450 for Xception, 0.9449 for F3Net, 0.9434 for FFD and 0.9431 for CORE; ten of the fifteen detectors average above 0.93 and only the two MesoNet variants, CNN-Aug and Capsule fall below 0.85. The best single cell is UCF's 0.9896 on the FaceSwap subset. The paper's own comment is that "Naive Detectors (e.g., Xception and EfficientB4), which essentially rely on a straightforward CNN classifier, register high AUC values that are comparable to more sophisticated algorithms".

Two of the six within-domain sets are harder than the rest. FaceForensics++ at the c40 compression level pulls every detector down, Xception to 0.8261 and UCF to 0.8399, and it is the lowest within-domain cell for twelve of the fifteen detectors; the NeuralTextures subset is the lowest of the four manipulation methods for every row. Both effects come from the test material, not from new generators. The frame-level AUC table by test set carries every cell with its conditions.

03How accurate are they on fakes from a dataset they never saw?

Lower, and the spread widens. Across the eight cross-domain columns of the same table, the lowest cells are Meso4 on DFD at 0.5481, Face X-ray on DeeperForensics-1.0 at 0.5531 and FWA on FaceShifter at 0.5551, all within a few points of chance; the highest is UCF on UADFV at 0.9528. Cross-domain averages run from Meso4's 0.6551 to SPSL's 0.7875, with Xception at 0.7718. For Xception the drop from its seen score is 0.9637 on FaceForensics++ c23 to 0.7365 on Celeb-DF v2 and 0.7077 on DFDC.

Which unseen set is hardest depends on the detector. UADFV, a small early set, is the easiest unseen set for eleven of the fifteen and DeeperForensics-1.0 for the other four; FaceShifter is the hardest for eleven. The ranking of detectors changes too: the within-domain leader UCF leads nine of the sixteen columns, but SPSL takes both Celeb-DF columns and the cross-domain average, and Face X-ray leads FaceShifter at 0.6553 while trailing on DeeperForensics at 0.5531. The cross-dataset generalisation page explains why the traces a detector learns on one set fail to transfer.

04How accurate are they on fakes that actually circulated?

Near chance for open-source models off the shelf. Deepfake-Eval-2024 collected 45 hours of video, 56.5 hours of audio and 1,975 images flagged by users in 2024 from 88 websites in 52 languages, and ran models that had been chosen before testing. Its Table 3 gives video AUC of 0.63 for GenConViT, 0.50 for FTCN and 0.51 for Styleflow, against 0.96, 0.87 and 0.95 on the benchmarks each was published on. Averaged over models and modalities, the authors report a drop in AUC of 50 percent for video, 48 percent for audio and 45 percent for images, and note that "many off-the-shelf models have an AUC close to 0.5, the same as random guessing".

Two further rows qualify the picture. Commercial video models, tested anonymously with vendors blind to the data, ranged from 0.64 to 0.79 AUC, and the paper states that no commercial model "had an accuracy of 90% or above" (Table 5). Fine-tuning GenConViT on 60 percent of the new set and testing on the other 40 percent raised its AUC to 0.82 (Table 4), a different protocol, since the model had then seen the distribution. The in-the-wild benchmarks page covers the collection method and the audio and image results.

05Why do the three answers differ so much?

The three conditions remove the detector's advantages one at a time. On seen fakes the test split shares generators, source footage and compression with the training split, so traces specific to the set still work. On unseen fakes the generator changes, and DeepfakeBench's cross-manipulation experiments name the blending-based detectors Face X-ray and DSP-FWA as "the exception" that transfers between manipulation methods. In the wild the generator, the compression and the real footage all change at once, and Deepfake-Eval-2024 reports that off-the-shelf GenConViT and FTCN are 21.3 percent less accurate on videos that appear diffusion generated.

The numbers also differ in kind. DeepfakeBench reports frame-level AUC on curated sets with known labels; Deepfake-Eval-2024 reports video AUC on found media labelled by a team of seven, with a 10 percent double review showing 6.6 percent disagreement on video. A 0.96 from the first and a 0.63 from the second are not two measurements of one quantity but two quantities, and quoting either without its condition is the error this page exists to prevent. NIST's face recognition evaluations handle the same problem by reporting an error rate at a fixed operating point on a named set, the form described on the face recognition accuracy page; and none of the deepfake laws now in force sets an accuracy requirement that would force detection papers to do likewise.

No single benchmark score predicts how a detector performs on unseen manipulations.