Celeb-DF v1 vs v2: what changed and why detector scores differ

This page is part of the deepfake detection hub. It explains the two versions and the gap between their scores; the dataset itself is described on the Celeb-DF dataset page.

01What is the difference between Celeb-DF v1 and v2?

The versions differ in how many videos and people they contain. The Celeb-DF repository describes v1 as 408 original videos collected from YouTube and 795 DeepFake videos synthesized from them, and v2 as 590 original videos, 5,639 DeepFake videos and 300 additional YouTube videos (GitHub repository). DeepfakeBench's dataset table gives the totals as 1,203 videos for v1 and 6,229 for v2, with 59 subjects in v2 and none listed for v1 (DeepfakeBench README).

The v1 figures come from the earlier version of the paper (arXiv v2, 2 October 2019), which describes 158 videos of 13 celebrities plus 250 videos each with a different subject. The final paper (arXiv v4, March 2020) describes the larger set: 590 real videos, 5,639 DeepFake videos and "over two million video frames" (arXiv 1909.12962).

02Did the way the fakes were made change?

The refinements to the synthesis method are described in both versions of the paper. The authors list a face generated at 256 by 256 pixels where the original algorithm produced 64 by 64, colour augmentation to reduce mismatch, a smoother non-rectangular mask, and less temporal flicker. The 2019 text that covers the 795-video set already names these four. What grew between releases is the number of videos and the people in them: the final paper says the fakes "are generated by swapping faces for each pair of the 59 subjects".

The authors' aim was stated in the abstract: existing sets "suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet". Neither version is a collection of real circulating fakes; both are made by the authors' own synthesis method from YouTube interviews.

03How much lower do detectors score on v2?

Every one of the 15 detectors scores lower on v2. DeepfakeBench trains each on FaceForensics++ c23 and reports frame-level AUC on 14 test sets including both Celeb-DF versions (arXiv 2307.01426). Subtracting the README's v2 column from its v1 column gives drops between 0.0266 (UCF) and 0.1267 (Meso4), a middle value of 0.0405 and a mean of 0.0504; the mean AUC over the 15 detectors goes from 0.7713 on v1 to 0.7209 on v2. These differences are arithmetic on the published table, not figures the paper states.

Two rows stand out. FWA falls by 0.1217, from 0.7897 to 0.6680, the second largest drop. SPSL has the highest score on both versions, 0.8150 and 0.7650, so the order at the top does not change even though the level does. Xception, the plain baseline, falls from 0.7794 to 0.7365. The wider pattern across test sets is on the cross-dataset generalisation page.

04Why do Celeb-DF scores differ between sources?

Four things differ, and a number is only comparable when all four match.

• Version. A paper that says "Celeb-DF" without a version may have used either release, or the full 590 and 5,639 set. DeepfakeBench lists CelebDF-v1 and CelebDF-v2 as separate datasets with separate columns.
• Test list. The v2 repository ships a list of 518 test videos. DeepfakeBench reads each version's own testing list when it builds its splits, so a result on the test list and a result on all videos are different measurements.
• Protocol. DeepfakeBench trains every detector itself with 32 frames per video and reports frame-level AUC. The Celeb-DF paper ran published pre-trained models and, because the videos are compressed, evaluated key frames only. Its Table 4 puts Xception-c23 at 65.3 percent AUC on Celeb-DF; DeepfakeBench puts Xception at 0.7365.
• Date of the paper. In the 2019 text covering the 795 videos, FWA scored 53.8 percent AUC on Celeb-DF; in the 2020 text covering 5,639 it scored 56.9. The detectors and test videos differ between the two papers, so the change cannot be put down to the dataset alone.

The DeepfakeBench page describes the protocol behind the table used here, and the FaceForensics++ page the training set.

05Which version should a result be compared with?

Compare a score only with scores on the same version, the same test list and the same metric. For a number quoted as "Celeb-DF" with no version, read the paper's data section before using it. Celeb-DF scores also describe only one kind of fake: in the same DeepfakeBench table Xception scores 0.7077 on the Deepfake Detection Challenge set, against 0.7365 on Celeb-DF v2.