DF40: forty deepfake techniques in one detection dataset

This page is the dataset entry for DF40 in the deepfake detection hub. It sits beside the datasets table, which places DF40 next to FaceForensics++, Celeb-DF and the other public sets, and it reads the paper's tables rather than repeating them in full.

01Why was DF40 built?

The paper names three gaps in the datasets that detection research had relied on. Diversity: FaceForensics++ contains four manipulation methods, two swaps and two reenactments. Realism: "the dominated training dataset, FF++, contains out-of-date forgery techniques from the past four years", so a detector tuned to them has no guarantee against current generators. Protocol: "most detection works perform evaluations on one type", training and testing on face swaps only, which cannot show whether a detector generalises across kinds (Yan et al., 2024).

The response was scale on the method axis rather than the footage axis. Where earlier sets took one or a few generators and many videos, DF40 takes 40 generators, which the abstract describes as "10 times larger than FF++" in method count, and applies them to footage the field already knew. The paper's Table 1 compares it with earlier sets by the number of methods in each of four categories, and gives DF40's own totals as over 0.1 million fake videos and over 1 million fake images, with pretrained detector weights released.

02What are the four categories and forty methods?

DF40 follows a four-way classification: "face-swapping (FS), face-reenactment (FR), entire face synthesis (EFS), and face editing (FE)". The Table 1 caption gives the split: "10 FS methods, 13 FR methods, 12 EFS methods, and 5 FE methods". The repository's method table lists them in that order, numbered 1 to 40 (DF40 repository). The kinds of face deepfake page explains what each category changes in the image, and the deepfake meaning and origin page shows how a word that began as one method name came to cover all four.

The paper splits face swapping further into a DF-family, which "involves creating a mask around the facial region ... and blending the generated deepfake face back into the background image using that mask", and an FS-family that generates the whole image. It notes that the familiar datasets, FaceForensics++ DeepFakes, Celeb-DF and DFDC, all belong to the DF-family, so a detector that has only seen those has only seen blended swaps.

03Which footage are the fakes made from?

Most of the 40 methods were run on two sources of real video: the FaceForensics++ originals and the Celeb-DF originals, which the paper calls the FF domain and the CDF domain. The repository states that "most methods (31) are generated under the FF++ and CDF domains". Pairing the same method with two domains is the point of the design: it lets the authors hold the forgery constant and change the footage, or hold the footage constant and change the forgery.

Nine further sets are test only and act as an unknown domain, built on CelebA, FFHQ, VFHQ and UADFV footage, "simulating the domain shift of real people distributions". Two of the most used commercial-grade generators, HeyGen and DeepFaceLab, appear in small numbers, 50 and 100 items, because they are one-to-one methods and "their implementation requires significantly more time and resources". Both are therefore evaluation sets rather than training sets within DF40.

04How were the detectors evaluated?

The experimental pipeline follows DeepfakeBench: "all pre-processing and training codebases in this work adhere to DeepfakeBench to align the standardized settings", and the primary metric is frame-level AUC, with video-level AUC reported for the one video model. The eight detectors are the Xception baseline; four detectors that use Xception as their backbone, SRM, SPSL, RECCE and RFM; the blending-based SBI, which trains on pseudo-fakes made by blending real images and is not retrained on DF40; CLIP, a pretrained vision-language backbone fine-tuned for the task; and the I3D video model, which can only be run on the swap and reenactment data because the synthesis and editing sets are still images. One procedural difference from DeepfakeBench is stated: DF40 uses a validation set to select the checkpoint that is then tested everywhere.

The four protocols are "cross-forgery evaluation (Protocol-1), cross-domain evaluation (Protocol-2), toward unknown forgery or domain evaluation (Protocol-3), and One-Verse-All (OvA) evaluation (Protocol-4)". Protocol 1 trains on one category in the FF domain and tests on the other categories in the same domain. Protocol 2 trains in the FF domain and tests on the CDF domain. Protocol 3 tests on the nine unknown-domain sets. Protocol 4 trains Xception on a single method and tests it on every other method, producing the heatmaps in the paper's Figure 3, where "white means 0.5 AUC (by chance)". The DeepfakeBench page describes the shared pipeline these numbers rest on, and the ROC curve for deepfake detection page explains why 0.5 is the chance line.

05What did the tables show?

The paper lists seven findings; the tables behind the first four are summarised here with the Xception baseline, since it appears in every protocol. Under Protocol 1 (Table 3, FF domain, frame-level AUC), Xception trained on face swaps scored 0.991 on swaps, 0.892 on reenactments and 0.810 on entire synthesis; trained on reenactments it scored 0.838, 0.996 and 0.670; trained on synthesis it scored 0.665, 0.807 and 0.999. The drop is asymmetric, which is Finding 1: "model training on FS yields higher results (around 0.8) on EFS, while training on EFS only achieves about 0.6 on FS".

Under Protocol 2 (Table 4), the same Xception model trained on FF-domain swaps and tested on the CDF domain scored 0.922 on swaps, 0.657 on reenactments and 0.642 on synthesis, an average of 0.740. Finding 4 reads those numbers: "forgery methods and data domains together contribute to discriminative forgery artifacts", with the fall to 0.657 and 0.642 described as nearly a 20 percent drop against the single-change cases. Under Protocol 3 (Table 5), Xception trained on FF-domain swaps averaged 0.546 across the nine unknown sets, scoring 0.882 on DeepFaceLab but 0.394 on HeyGen, below chance; CLIP averaged 0.692, with 0.930 on DeepFaceLab and 0.539 on HeyGen.

Two findings concern the detectors rather than the data. Finding 2: "existing well-designed (SoTA) detectors may not have obvious advantages over the baseline", since SRM, SPSL, RECCE and RFM scored close to plain Xception. Finding 3: "CLIP excels in deepfake detection than other baselines (such as Xception)", across every protocol. The published results table on this site reproduces DeepfakeBench rows and notes the DF40 protocols separately, because a DF40 AUC is measured on a different test set and with checkpoint selection on a validation split, and so is not directly comparable with a DeepfakeBench row for the same model name.

06What are DF40's limits?

The authors state one themselves: "one limitation is the lack of comprehensive analysis for video-level detectors", since only I3D was run and only on the two video categories. Two more follow from the design. The real footage is still FaceForensics++ and Celeb-DF, so the fakes inherit those sets' interview framing and YouTube provenance, and a DF40 score remains a laboratory number; the deepfake detection in the wild page carries the figures for fakes that circulated, and the documented deepfake incidents page shows that the fakes in recorded fraud cases were pre-recorded clips built from public footage, a condition no protocol here reproduces. And method diversity is a moving target: the latest method in the set is dated 2024, and the gap the paper describes between FaceForensics++ and current generators will reopen for DF40 on the same timetable.

The dataset is released under a CC BY-NC 4.0 licence, with download by application form and detector checkpoints on a shared drive, and the paper was accepted to the NeurIPS 2024 Datasets and Benchmarks track. No single benchmark score predicts how a detector performs on unseen manipulations; DF40's contribution is to make "unseen" measurable along two axes at once, forgery and footage.