Public deepfake datasets compared: FaceForensics++, Celeb-DF, DFDC, DF40 and more

ROWThe datasets in the table

01How to read the table

This is the dataset reference for the deepfake detection benchmark hub. The published detector results page reports what detectors score on these sets; this page is about the sets themselves.

Smaller or older sets that still appear in papers but are not in the table: UADFV (49 real and 49 fake videos, 49 subjects) and DeepFake-TIMIT (320 real and 640 fake videos, 43 subjects), as listed in the ForgeryNet paper's comparison table. No file, frame or face from any of these sets is hosted on this site.

Lab sets versus in-the-wild sets

Seven of the nine sets were made in a lab: the authors took real videos and ran manipulation software over them. That gives clean labels and large numbers, and it means every fake in the set was made by a method the authors chose, at the quality their pipeline produced. WildDeepfake and Deepfake-Eval-2024 are the opposite kind: fakes collected from the internet, made by unknown people with unknown tools. Their labels are harder to guarantee and their numbers are smaller, but they measure the thing a deployed detector will actually meet. The in the wild page covers what happens to detector scores on them.

Why counts are not comparable across sets

"Videos" means different things. FF++ counts 1,000 source clips and 1,000 manipulated outputs per method. DFDC counts short clips cut from longer recordings of the same actors. WildDeepfake counts face sequences extracted from 707 videos, which is why its headline is 1.18 million face images. A set with more rows is not a harder set, and a set with more methods is not automatically a more diverse one if the methods are all face swaps. Read the methods column with the counts.

The rights column

Three of the lab sets (DFD, DeeperForensics-1.0, DFDC) were filmed with people who consented; DFDC's were paid actors. FF++ and Celeb-DF used YouTube footage of people who did not, and Celeb-DF's documentation offers removal on request. The internet-collected sets by construction contain people who never agreed to anything. The same tension runs through face recognition datasets, including the one that used to live on this domain; the face datasets and consent page follows it.

Which dataset should you test on?

Train on one, test on several others, and report the unseen scores first. If the detector will face recent generators, include DF40 or an in-the-wild set; FF++ alone tests against 2019 methods. If false positives on real people matter, use a set whose real videos resemble your deployment, not FF++'s YouTube interviews. The cross-dataset generalisation page shows how large the published drops are, and the evaluate a detector protocol turns this into steps.

02Method and sources

Rows are built from each dataset's paper on arXiv or its official repository README, read on the date shown. Where sources disagree on a count, the table shows the more specific figure and names the disagreement here: DFDC's paper says "over 100,000 total clips"; DeepfakeBench's table gives 23,654 real and 104,500 fake; ForgeryNet's comparison table gives 23,564 real and 104,500 fake. The real count differs by 90 between the two secondary tables and should be read as about 23,600. Nothing is estimated. Changes to a dataset after this date are not reflected until the next check; the date column is the promise.

FAQQuestions

RELRelated reading

• D.1.2FaceForensics++: the four manipulation methods, compression levels and how detectors are scored on itFaceForensics++ explained: four manipulation methods, the raw, c23 and c40 compression levels, the benchmark protocol, and how detector scores are read.
• D.1.3Celeb-DF: why it was built to be harder, and what cross-dataset scores on it showCeleb-DF explained: why it was built to remove the visual artefacts of earlier sets, what it contains, and what detectors trained elsewhere score on it.
• D.1.4The Deepfake Detection Challenge (DFDC): the dataset, the Kaggle contest and the black-box gapThe Deepfake Detection Challenge explained: how the DFDC dataset was built with consenting actors, how the contest was scored, and the black box gap.
• D.1.5DeepfakeBench: a shared codebase for comparing detectors, and how to read its tablesDeepfakeBench explained: what the shared codebase standardises across deepfake detectors, how its train on one, test on many table works, how to read it.
• D.2.4Deepfake detection in the wild: what benchmarks built from real circulating fakes foundDeepfake detection in the wild: what benchmarks built from real circulating fakes, such as Deepfake-Eval-2024, found about accuracy versus lab sets.
• D.2.3Why deepfake detectors fail on new fakes: cross-dataset generalisation explainedWhy deepfake detectors fail on new fakes: what cross-dataset evaluation measures, how large the published drops are on unseen sets, and what helps.
• D.2.1Deepfake detector results from published papers: in-dataset and cross-dataset scores, with datesDeepfake detector results from published papers and official leaderboards: in-dataset and cross-dataset scores by model, with test set, metric and date.

SRCSources