The Deepfake Detection Challenge (DFDC): the dataset, the Kaggle contest and the black-box gap

This is the DFDC entry in this site's dataset comparison and part of the deepfake detection hub. It is also, with the ICCV 2021 masked face recognition challenge that used to be documented on this domain, one of two competition write-ups here, and the two make a useful pair: both were judged on hidden test data under rules that the entrants could not tune to.

01What DFDC was

The dataset paper, "The DeepFake Detection Challenge (DFDC) Dataset" by Dolhansky, Bitton, Pflaum, Lu, Howes, Wang and Canton Ferrer (arXiv 2006.07397, 2020), describes both halves. The dataset: "an extremely large face swap video dataset to enable the training of detection models", which the authors describe as "by far the largest currently and publicly available face swap video dataset, with over 100,000 total clips sourced from 3,426 paid actors, produced with several Deepfake, GAN-based, and non-learned methods". The contest: "the accompanying DeepFake Detection Challenge (DFDC) Kaggle competition", whose top submissions the paper analyses in detail.

Secondary tables give the finer counts. DeepfakeBench lists DFDC at 23,654 real and 104,500 fake videos (128,154 total), 960 subjects in the released portion it uses, 8 synthesis methods and 19 perturbations; it also lists the earlier DFDC Preview release at 1,131 real and 4,119 fake videos from 66 subjects with 2 methods (DeepfakeBench repository). The ForgeryNet paper's comparison table gives 23,564 real clips, 90 fewer; this site reads the real count as about 23,600 and names both sources.

02How the dataset was built

The most important sentence in the paper, for anyone comparing datasets, is about the people in it: "all recorded subjects agreed to participate in and have their likenesses modified during the construction of the face-swapped dataset". The 3,426 actors were paid and recorded for the purpose. That puts DFDC on a different footing from FaceForensics++ and Celeb-DF, whose source footage came from YouTube, and it is why DFDC is the set to use when a project needs consented data. The face datasets and consent page follows that distinction across face datasets generally.

Diversity of forgery was the other design goal. Rather than one swap method, the authors used several Deepfake-style, GAN-based and non-learned methods, and applied perturbations (the DeepfakeBench table counts 19 kinds) so that detectors could not rely on the artefacts of a single pipeline or a single video quality.

03How the contest was scored

The Kaggle competition asked entrants to submit a probability that each video was fake. Scoring used a log-loss style metric on a public test set during the competition and a private, black-box test set for the final ranking. The paper's detailed analysis of the top entries, including how they fared on the black-box set, is the authoritative account; this site does not reproduce the leaderboard, and it does not quote the winning scores until they have been read from the paper's tables and dated.

04The black-box gap

The reason DFDC is cited in every discussion of generalisation is what happened between the public and private test sets. Entrants could see and tune to the public set; the black-box set contained material they had not seen. Scores fell between the two, and the paper is candid that "Deepfake detection is extremely difficult and still an unsolved problem". The same paper reports the counterweight: a detection model "trained only on the DFDC can generalize to real in-the-wild Deepfake videos, and such a model can be a valuable analysis tool when analyzing potentially Deepfaked videos".

Both halves of that finding hold up in later work. The cross-dataset generalisation page shows the drop that every detector suffers on unseen sets, and the in the wild page shows that detectors trained on large consented sets do transfer something, if not enough.

05DFDC in later benchmarks

DFDC and DFDC Preview are standard unseen test sets. In the DeepfakeBench evaluation (detectors trained on FaceForensics++ c23, frame-level AUC, 2023 table) the Xception baseline scores 0.7077 on DFDC and 0.7374 on DFDC Preview; the best DFDC score in that table is UCF at 0.7191 and the best DFDC Preview score is also UCF at 0.7594 (DeepfakeBench results table). DFDC is, with FaceShifter, among the hardest of the lab sets in that table for detectors that have never seen it. The full table with dates is on the results page.

06Access

The paper gives the dataset site and the Kaggle competition page as sources for the training, validation and testing corpora. Availability is the publishers' decision; this site hosts nothing from the set and links the paper rather than a download.