Unseen deepfake generators: what DF40 found about detectors on new fakes

This page belongs to the deepfake detection hub and reads the protocol tables of the DF40 dataset for one question: what happens to a detector when the generator changes. The dataset page describes the 40 techniques; the figures here are the published AUC values with their training and test conditions.

01What does "unseen generator" mean in DF40?

A detector sees one set of generators during training and is scored on fakes from generators it never saw. DF40 makes the change explicit through four protocols: cross-forgery (Protocol 1, new forgery types on the same footage), cross-domain (Protocol 2, the same forgery types on different footage), unknown forgery and domain together (Protocol 3) and one-versus-all (Protocol 4, train on one method and test on each of the others) (arXiv 2406.13495). The paper groups its 40 methods into four types: face swapping (FS, 10 methods), face reenactment (FR, 13), entire face synthesis (EFS, 12) and face editing (FE, 5), and most of them were run on the FaceForensics++ and Celeb-DF footage.

The eight detectors are Xception, the classical baseline; SRM, SPSL, RECCE and RFM, four later detectors that keep Xception as their backbone; SBI, a blending-based detector; CLIP, a pre-trained vision model fine-tuned for the task; and I3D, a video model. Preprocessing and training follow DeepfakeBench, and the primary metric is frame-level AUC, so the numbers are comparable with the DeepfakeBench tables built under the same settings.

02Did purpose-built detectors beat the baseline?

Not on DF40. Table 3 scores detectors trained on FS fakes from the FaceForensics++ footage and tested on each forgery type in the same footage. Xception averaged 0.898 AUC (0.991 on FS, 0.892 on FR, 0.810 on EFS). SRM averaged 0.853, SPSL 0.857, RECCE 0.868 and RFM 0.899; CLIP averaged 0.914. The paper's wording is that the four Xception-backbone detectors "achieve results similar to the simple baseline Xception" and that they "might only learn sub-optimal forgery features and lack clear advantages over the baseline, although these SoTAs show higher results on previous datasets".

Three of the four purpose-built detectors scored below the baseline on the hardest column, EFS, and the best of them, RFM, beat Xception's average by 0.001. The gap between the baseline and the pre-trained CLIP, 0.016, is larger than the spread among the five Xception-family models.

03What happens when the footage changes too?

Table 4 keeps the same trained models and tests them on fakes made from Celeb-DF footage. Xception's average fell from 0.898 to 0.740, with 0.922 on FS, 0.657 on FR and 0.642 on EFS. The purpose-built detectors fell with it: SRM to 0.714, SPSL to 0.747, RECCE to 0.723 and RFM to 0.735. CLIP fell to 0.814. The paper's fourth finding reads the drop: "forgery methods and data domains together contribute to discriminative forgery artifacts", and it describes the FR and EFS results on Celeb-DF footage as "nearly a 20% drop" from the same-footage figures.

Table 5 is the hardest protocol, nine test-only sets on footage and generators the detectors never saw. Trained on FS (FF), Xception averaged 0.546, SRM 0.564, SPSL 0.565, RECCE 0.536 and RFM 0.648; CLIP averaged 0.692. Within those averages the spread by generator is wide. Xception scored 0.882 on DeepFaceLab swaps, a method of the same family as its training data, and 0.394 on HeyGen reenactments, below the 0.5 that chance would give. CLIP scored 0.930 and 0.539 on the same two sets.

04Why does the forgery type matter as much as the generator?

DF40's first finding is an asymmetry: "model training on FS yields higher results (around 0.8) on EFS, while training on EFS only achieves about 0.6 on FS". Table 3 shows it. Xception trained on EFS scored 0.999 on EFS but 0.665 on FS and 0.807 on FR; trained on FS it scored 0.810 on EFS. The authors' explanation is about where the fake pixels are: a swap can leave artefacts in a face region or across the frame, while a synthesised image is fake everywhere, so a detector trained on whole-frame fakes "might be biased to 'expect' artifacts across the entire image".

The fifth finding adds a second axis. Most reenactment methods "share transferable patterns" so that a detector trained on one FR method does well on others, with Wav2Lip, which edits only the mouth, as the exception. Where a new generator falls in that taxonomy predicts the drop better than its release date does. The dataset page lists all 40 methods with their type and footage.

05Does pre-training help against new generators?

In DF40 it did, more than architecture did. Finding 3 states that CLIP, "both base and large versions", outperformed the other detectors and the baseline "across all scenarios", and Finding 7 reports that CLIP-large "shows the potential to generalize to some non-face deepfakes when trained only on face data". The gap was largest where the task was hardest: 0.016 over Xception on same-footage averages, 0.074 on Celeb-DF footage and 0.146 on the unknown domains.

The paper does not claim the problem is solved. CLIP's unknown-domain average of 0.692 is far from its 0.914 on familiar footage, and its score on HeyGen, 0.539, is near chance. The authors also note "the lack of comprehensive analysis for video-level detectors" as a limitation, since I3D is the only video model tested.

06What does this mean for a published detector score?

A detector's score on the benchmark it was built for says little about a generator released after that benchmark. DF40's authors motivate the dataset by noting that FaceForensics++ "contains out-of-date forgery techniques from the past four years", and their Table 5 shows what that costs: five detectors that score 0.99 on their training generators average 0.54 to 0.65 on nine unseen sets. The cross-dataset generalisation page documents the same drop between older benchmarks, and the same shape appears in other media: in the ASVspoof 2021 results, speech detectors that scored below 1 percent EER on familiar sources exceeded 15 percent on new ones, and in the Deepfake Detection Challenge the winning video model's log loss rose from 0.1983 on challenge footage to 0.6605 on organic internet video.

No single benchmark score predicts how a detector performs on manipulations it was not trained on. That is also why the official fraud statistics discussed on the deepfake fraud statistics page record cases and not detection rates: no regulator has a number for how often a new generator slips past a detector, and the research tables suggest the answer changes with every release.