Why deepfake detectors fail on new fakes: cross-dataset generalisation explained

This page is part of the deepfake detection benchmark hub. It explains what the cross-dataset protocol measures, puts numbers on the drop from the one shared benchmark that reports them consistently, and summarises what the papers say reduces it. The full per-model table is on the published results page.

01What cross-dataset evaluation measures

A detector learns from examples. If every fake it sees in training was made by the four methods in FaceForensics++, at FaceForensics++'s compression, on FaceForensics++'s YouTube interview footage, then what it learns is some mixture of three things: general signs that a face has been altered, the specific fingerprints of those four tools, and the statistics of that footage. Only the first of those transfers to a new fake. In-dataset testing, where the test split comes from the same dataset, cannot tell the three apart, because all three are present in the test images too.

Cross-dataset testing removes the second and third. Train on FaceForensics++, test on Celeb-DF, and the detector meets a different face-swap method, different compression, different people and different cameras. Whatever accuracy survives is accuracy that came from general cues. The datasets page explains why Celeb-DF and DFDC became the standard unseen sets: Celeb-DF was built specifically to remove the visual artefacts that made earlier fakes easy, and DFDC was built at scale with many methods and perturbations.

02The published drop

DeepfakeBench, the shared evaluation codebase from the Chinese University of Hong Kong, Shenzhen (NeurIPS 2023), retrained a set of detectors under one preprocessing pipeline and one protocol, trained on FaceForensics++ at c23 compression, and reported frame-level AUC on the training set and on unseen sets. Its results table, cited in the repository from the paper, is the cleanest published picture of the drop because every method was handled the same way.

Three rows from that table, in-dataset versus unseen:

• Xception, the standard baseline: FaceForensics++ c23 0.9637; Celeb-DF v1 0.7794; Celeb-DF v2 0.7365; DFDC 0.7077; DFDC Preview 0.7374; FaceShifter 0.6249.
• UCF, the best in-dataset method in the table: FaceForensics++ c23 0.9705; Celeb-DF v2 0.7527; DFDC 0.7191; cross-dataset average 0.7801.
• SPSL, a frequency-domain method and the best cross-dataset average in the table: FaceForensics++ c23 0.9610; Celeb-DF v1 0.8150; Celeb-DF v2 0.7650; cross-dataset average 0.7875.

Read across, not down. The method that scores highest on its own training set (UCF, 0.9705) is not the one that generalises best (SPSL, 0.7875 average). The difference between the best and worst serious method on FaceForensics++ is about one point of AUC; the difference between in-dataset and unseen for the same method is twenty points or more. Choosing a detector by its in-dataset score chooses on the smaller signal. Source for every figure: the DeepfakeBench results table, from the paper at arXiv 2307.01426, checked 2026-11-08; the repository notes that an expanded v2 table is pending, so these are the 2023 numbers.

The same shape appears in every dataset paper that tests existing detectors on its new set. WildDeepfake's authors ran ten image-level baseline detectors, several of which score above 0.99 accuracy on DeepFake-TIMIT or high-quality FaceForensics++ (Deepfake), and found none above 70% on their internet-collected fakes (arXiv 2101.01456). Deepfake-Eval-2024 reports open-source detectors' AUC falling by 50% for video and 45% for images compared with previous benchmarks (arXiv 2503.02857). Those are the in-the-wild version of the same problem.

03Why it happens

Method-specific artefacts

Each manipulation method leaves its own traces: blending seams at a particular scale, a characteristic texture in the swapped region, frequency patterns from a particular upsampling layer. A detector with no reason to prefer general cues will latch onto whichever trace is easiest in its training data. When the test fake was made by a different method, that trace is absent. The DF40 paper puts the problem at the dataset level: FaceForensics++ "contains out-of-date forgery techniques from the past four years", so a detector tuned to them is tuned to history (arXiv 2406.13495).

Compression and quality

FaceForensics++ ships at raw, c23 and c40. Detectors trained at one level and tested at another lose accuracy, because compression destroys exactly the fine-grained inconsistencies many detectors rely on; the Xception row above falls from 0.9637 at c23 to 0.8261 at c40 within the same dataset. Internet video is recompressed at every upload. A fake that circulated has been through more compression than any lab set.

Source footage and identity

If the real and fake videos in a training set come from the same pool of faces, a detector can learn the faces rather than the forgery. If the real videos share a camera, a lighting setup or a platform, it can learn those. Celeb-DF's design, swapping faces between public figures' YouTube clips, and DFDC's, with 3,426 actors and 19 perturbations, were both attempts to break these shortcuts.

04What helps, according to the papers

The methods that top the cross-dataset columns in DeepfakeBench share a theme: they push the detector away from dataset-specific traces. Frequency-domain inputs (SPSL and F3Net in the table) look at spectral patterns that different generators share. Self-blending and similar augmentations (SBI, which DeepfakeBench v2 implements) make training fakes out of real images with no generator at all, so there is no tool fingerprint to memorise. Methods that explicitly separate common forgery features from specific ones (UCF) aim at the same target from the representation side. None closes the gap; the best cross-dataset average in the 2023 table is 0.7875, against in-dataset scores near 0.97.

This site does not rank these approaches or recommend one. The published results page lists what each scored, where, and when; the DeepfakeBench page explains how to read the shared codebase's tables.

05In the wild is the harder version

Cross-dataset testing on lab sets is a controlled approximation. The fakes in Celeb-DF were still made by researchers with one method. Testing on fakes that actually circulated, as WildDeepfake and Deepfake-Eval-2024 do, removes the last control and the scores fall further. The in the wild page carries those numbers.

06Choosing a test set

For anyone evaluating a detector, the rule that follows is short: report unseen-set scores first, name the training set and the test set in the same sentence, and include at least one set built from recent methods or from circulating fakes. The datasets page lists the options with their access routes, and the bridge page shows that face recognition met the same problem as the gap between benchmark accuracy and field accuracy.