Celeb-DF: why it was built to be harder, and what cross-dataset scores on it show

This is the Celeb-DF entry in this site's dataset comparison, part of the deepfake detection hub. Everything below is taken from the dataset's paper and repository, with dates.

01What Celeb-DF is

The paper, "Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics" by Li, Yang, Sun, Qi and Lyu (arXiv 1909.12962, CVPR 2020), presents the second version of the set. The repository summarises it: "590 original videos collected from YouTube with subjects of different ages, ethic groups and genders, and 5639 corresponding DeepFake videos" (Celeb-DF repository). The folder layout adds 300 additional real YouTube videos and a test list of 518 videos. The first version, Celeb-DF v1, had 795 deepfake videos and is still available alongside.

The 590 originals are of 59 people, according to the dataset tables in DeepfakeBench and the ForgeryNet paper. They are public figures, which is where the name comes from and where the consent question sits (below).

02Why earlier datasets were too easy

The authors' motivation is the first sentence of the abstract's second half: existing sets had "low visual quality" and did not look like the fakes people were actually sharing. Early public deepfakes were made quickly, with visible seams, colour mismatches between the swapped face and the surrounding skin, flicker between frames and low-resolution swapped regions. Detectors trained on those sets learned to find those flaws, which made them look excellent on the sets and useless the moment a better fake appeared.

Celeb-DF's answer was an improved synthesis process aimed at removing those visible flaws, and a "comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF", in the abstract's words. The repository's own chart compares average AUC of detection methods across datasets, with Celeb-DF the lowest; the paper has the figures.

03What the dataset contains

• Celeb-real: 590 videos of 59 public figures, downloaded from YouTube.
• YouTube-real: 300 further real videos, also from YouTube.
• Celeb-synthesis: 5,639 face-swapped videos generated from the Celeb-real set.
• A list of 518 testing videos, so that published results use the same split.

One manipulation method is used throughout, so Celeb-DF measures a detector's response to a better face swap, not to a wider variety of forgeries. For method diversity the field uses DFDC, ForgeryNet or DF40; the datasets page compares them.

04Cross-dataset results on Celeb-DF

Because almost every detector is trained on FaceForensics++, "the Celeb-DF score" in a paper almost always means a cross-dataset score. In the DeepfakeBench evaluation (detectors trained on FaceForensics++ c23, frame-level AUC, 2023 table), the standard Xception baseline scores 0.7794 on Celeb-DF v1 and 0.7365 on Celeb-DF v2, against 0.9637 on its own training set. The best method on both Celeb-DF columns is SPSL, a frequency-domain detector, at 0.8150 (v1) and 0.7650 (v2). EfficientNet-B4 scores 0.7487 on v2, UCF 0.7527 (DeepfakeBench results table, from arXiv 2307.01426).

Two readings follow. First, v2 is harder than v1 for every method in the table, consistent with its purpose. Second, the spread between methods on Celeb-DF is wider than on FaceForensics++, so Celeb-DF is the more informative column for choosing between detectors. The full table and its dates are on the results page, and the cross-dataset generalisation page explains why the drop happens.

05Consent note

Celeb-DF's real videos are of named public figures who did not take part in the dataset. The repository's privacy statement says the originals "are not property of the authors or the authors' affiliated institutions" and offers to remove the information of anyone who feels uncomfortable about their identity appearing. That is a different footing from DFDC or DeeperForensics-1.0, whose subjects were recorded with agreement, and the same footing as the scraped face recognition sets covered on the face datasets and consent page. No frame from Celeb-DF, and no image of any of its subjects, appears on this site.