DFDC Preview: the 5,244-clip pilot before the Deepfake Detection Challenge

This page belongs to the deepfake detection hub and sits before the Deepfake Detection Challenge page in time: the full set of over 100,000 clips came eight months later, and most of the preview's design choices survived into it.

01What was released in October 2019?

The challenge itself was announced in September 2019, and the preview paper followed on 19 October 2019 with a second version four days later. Its abstract describes "a preview of the Deepfakes Detection Challenge (DFDC) dataset consisting of 5K videos featuring two facial modification algorithms", built from actors who "entered into an agreement to the use and manipulation of their likenesses". The paper's own comparison table lists the preview at 5,214 total videos with a tampered to original ratio of 1 : 0.28.

Two things made the release unusual for 2019. The footage was recorded for the purpose, not scraped: "No publicly available data or data from social media sites was used to create this dataset." And the organisers withheld the method names, writing that "no further details of the employed methods are disclosed to the participants", so that the detectors entered in the challenge could not be tuned to a known generator. The public dataset counts and licences table records the consent position of every later set against this one.

02How were the 5,244 clips made?

A set of 66 individuals was chosen from the crowdsourced actors and split into a training group and a test group, "to avoid cross-set face swaps". Swaps were computed between people of similar appearance, inferred from attributes such as skin tone, facial hair and glasses, and each pairwise model was run in both directions.

The two methods were not equal. Method A "does not produce sharp or believable face swaps if the subject's face is too close to the camera", so the organisers filtered every video by the ratio of face size to frame size and kept swaps only where that ratio was under 0.25. Method B "generally produces lower-quality swaps, but is similar to other off-the-shelf face swap algorithms". Every original and swapped video then lost its first five seconds, when actors were setting up cameras, and was cut into 15 second clips, "generally three clips per video if the length was over 50 seconds".

The test clips carry deliberate damage. For each target video in the test set, two of its three clips were degraded with one of three operations: frame rate reduced to 15 per second, resolution reduced to a quarter, or encoding quality lowered. "No video was subjected to more than one augmentation", and the third clip was left untouched. The training clips were released at their original quality. That is how the counts of 4,464 training and 780 test clips arise.

03Who is in the preview?

The paper gives a rough distribution of the actors: 74 percent female and 26 percent male; 68 percent Caucasian, 20 percent African-American, 9 percent east-Asian and 3 percent south-Asian. The organisers wrote that they would "keep working on improving the diversity towards the publication of the final DFDC dataset", and the full release drew on 3,426 actors (arXiv 2006.07397).

The consent model is the preview's lasting contribution. Of the five datasets in its comparison table, only DFDC Preview and the Google and Jigsaw Deep Fake Detection set were recorded with actors who agreed to be swapped; the other three took their source videos from YouTube.

04Why did the organisers invent weighted precision?

The preview paper's third section argues that every deepfake dataset has a ratio of fake to real far higher than any platform would see, so a precision measured on the dataset says little about precision in production. Writing the ratio of fakes to real videos as 1 : x in organic traffic and 1 : y in a dataset, the authors define a weighted precision wP = TP / (TP + αFP) with α = x / y, alongside ordinary recall R = TP / (TP + FN). A realistic prevalence would give α of about 4 × 10⁷; "because this preview dataset has few true negatives, any false positives will lead to large variations", so the paper reports wP with α = 100 and notes the constant "is subject to change for the final dataset".

Because false positives are weighted so heavily, wP is small, and the paper reports its logarithm: "a value of 0 is the maximum achievable value, but generally the log-weighted-precision is negative." It also reports log(wP) at three fixed recall levels, 0.1, 0.5 and 0.9, as a proxy for the cost of reviewing flagged videos. The full challenge dropped this metric and ranked by log loss instead, as the page on how the challenge scored log loss explains. The habit of fixing one error rate and reading the other is the same one NIST uses for face recognition, described on how to read a NIST report card.

05What did the three baselines score?

The paper tested three existing models, none trained on the preview: TamperNet, "a small DNN (6 convolutional layers plus a 1 fully connected layer)" trained to spot generic image edits such as pasted objects and added text, and two XceptionNet models from the FaceForensics++ release, one on the face crop and one on the full frame, with one frame sampled per second. Thresholds were tuned by cross-validation on the training set to maximise log(wP) at the wanted recall.

Read together, the rows say that the face-crop Xception model was the most precise but caught 8.4 percent of the fakes at its tuned threshold, and that every model's weighted precision collapsed as recall was pushed towards 0.9.

06How does DFDC Preview appear in later benchmarks?

DeepfakeBench (NeurIPS 2023) keeps the preview as a separate test set, DFDCP, and splits it by the organisers' method labels: "The dataset contains real videos and fake videos generated by two different methods: method A and method B. The train and test splits are determined based on the given method." Its dataset table lists the preview at 1,131 real and 4,119 fake videos, 5,250 in total, 66 subjects, 2 synthesis methods and 3 perturbations, with rights cleared (DeepfakeBench repository).

In DeepfakeBench Table 3, where every detector is trained on FaceForensics++ c23 and scored at frame level, the DFDCP column runs from Meso4 at 0.5994 to UCF at 0.7594, with the Xception baseline at 0.7374 and the phase-spectrum detector covered on the SPSL and the phase spectrum page at 0.7408 (arXiv 2307.01426). The same column is where two of the paper's side findings land: pre-training the Xception backbone was worth "about 10% in DFDCP", and compression and blur augmentations "may significantly degrade the performance of SPSL in the DFD and DFDCP datasets". Every value is listed, with its conditions, in the DeepfakeBench results by test set.

The preview is easier than the full challenge set for every detector in that table; Xception scores 0.7374 on DFDCP against 0.7077 on DFDC, which fits a set with two methods and three perturbations against one with eight methods and nineteen.

07Why numbers differ between sources

Three totals are in print for one release. The preview paper's Table 1 says 5,214 videos; the same paper's clip counts, 4,464 training and 780 test, add to 5,244; and DeepfakeBench's table says 5,250 from 1,131 real and 4,119 fake. The Celeb-DF paper's Table 1 and the WildDeepfake paper both give 1,131 real and 4,113 fake, which add to 5,244 (arXiv 1909.12962). This site headlines 5,244 because two sources agree on it and it matches the paper's own clip arithmetic, and quotes the 4,119 figure only where a page reports the DeepfakeBench table as published. None of the differences changes a benchmark result, because DeepfakeBench scores its own split.