DeeperForensics-1.0: the 60,000 video dataset with real-world perturbations

This page belongs to the deepfake detection hub and sits beside the other dataset entries in the timeline of deepfake datasets. It describes how the set was built, what its perturbations are, what its hidden test set measures, and how detectors trained elsewhere score on it.

01What is in DeeperForensics-1.0?

The dataset has two halves. The real half is studio footage: 100 actors, 55 men and 45 women from 26 countries aged 20 to 45, recorded by seven HD cameras at 1920 by 1080 under nine lighting conditions while performing eight named expressions, 53 blendshape expressions and head turns from minus 90 to plus 90 degrees. The paper reports over 50,000 source videos and 12.6 million frames from this collection alone (arXiv 2001.03024, section 3.1).

The fake half is 10,000 manipulated videos. The authors swapped the 100 actors' faces onto 1,000 driving videos taken from the FaceForensics++ source set using DF-VAE, a variational autoencoder built for the paper, to produce 1,000 raw swaps. They then expanded those into 5,000 videos with one perturbation each at a single level, 1,000 with a random perturbation at a random level, and 3,000 with a mixture of more than one perturbation. The DeepfakeBench dataset table records the result as 50,000 real, 10,000 fake, 100 subjects, one synthesis method and seven perturbations, with rights cleared (DeepfakeBench README); the dataset table with counts and licences on this site carries the same row.

02Why were the actors paid?

The earlier sets took their real footage from YouTube. DeeperForensics-1.0 took the position that found footage of people who never agreed to be swapped could not be used, even for research, and recorded its own. The 100 actors signed a formal agreement consenting to the use and manipulation of their faces, and the authors selected them to balance gender, age, skin tone and nationality, with four skin tones in a 1:1:1:1 ratio (section 3.1). The Deepfake Detection Challenge set, released the same year, took the same route with 3,426 actors; the DFDC page describes that set.

Consent changed what the set can contain. Because the real videos were shot indoors against a controlled background, the authors used only the cropped face region and treated the background as irrelevant, which is the opposite of an in-the-wild collection where the background and the re-encoding are part of what a detector has to cope with.

03What are the seven perturbations?

Table 2 of the paper lists them: colour saturation change, local block-wise distortion, colour contrast change, Gaussian blur, white Gaussian noise in the colour components, JPEG compression and video compression rate change. Each is applied at five intensity levels, so the set contains 35 distinct perturbations before mixtures are counted. The authors took the seven types from the image quality assessment literature as distortions "very common" in real-life video.

The point of the perturbations is measured in the paper's Table 5. Five video-level baselines trained and tested on the undistorted standard set score between 98.50 and 100.00 percent accuracy. The same models trained on the undistorted set and tested on single-level distortions fall to between 87.63 and 91.50 percent: C3D 87.63, TSN 91.50, I3D 90.75, ResNet+LSTM 90.63 and XceptionNet 88.38. Training on distorted data closes most of that gap, which is the paper's argument for including distortions in the training split rather than only in the test split.

04How do detectors score on it?

Two kinds of result exist. The first is the paper's own benchmark on its hidden test set: 400 videos made with several unknown face swapping methods, obscured with distortions, and kept only if they fooled at least 50 of 100 human observers. Trained on the DeeperForensics-1.0 standard set, the five baselines score 74.75 (C3D), 77.00 (TSN), 79.25 (I3D), 78.25 (ResNet+LSTM) and 77.00 percent (XceptionNet) on it. Trained instead on any one FaceForensics++ manipulation, the same models score between 52.13 and 63.63 percent on the hidden set, close to chance (Table 4).

The second kind comes from DeepfakeBench (NeurIPS 2023), which uses DeeperForensics-1.0 as one of eight unseen test sets for detectors trained on FaceForensics++ c23 (arXiv 2307.01426, Table 3). Frame-level AUC on the DF-1.0 column runs from 0.9334 for DSP-FWA down to 0.5531 for Face X-ray, with Xception at 0.8341 and EfficientNet-B4 at 0.8330. The frame-level AUC results by detector page lists every cell, and the cross-dataset grid by detector and test set sets the column beside the other seven.

05Why do weak detectors score well here?

The table has an unusual shape. Meso4, the weakest detector in DeepfakeBench overall with a cross-dataset average of 0.6551, scores 0.9113 on DeeperForensics-1.0, and DeeperForensics-1.0 is the best of the eight unseen sets for four detectors: Meso4, MesoInception, Capsule and DSP-FWA. For the other eleven, the easiest unseen set is UADFV. The dataset's construction bears on this: the 10,000 fakes come from one swapping method applied to the 1,000 FaceForensics++ driving videos those detectors trained on, while the 50,000 real clips are studio recordings, so the two halves differ in source footage as well as in manipulation. The authors themselves note that, as in FaceForensics++, the training and test splits of their standard set share a close distribution.

Face X-ray is the counter-example. DeepfakeBench describes it as a detector that "uses blended artifacts in forgeries" to generalise to unseen manipulations, and it scores 0.9872 on Face2Face within FaceForensics++ but 0.5531 here. DF-VAE adds a masked style matching module to fix the style mismatch between swapped face and frame and blurs its mask to avoid boundary artefacts (section 3.2), so the trace this detector looks for is the one the generator was built to suppress. The two results together show that a score on one unseen set measures the match between a detector's assumption and that set's generator, not generalisation in the abstract; the CNN detector comparison page makes the same point for the plain classifiers, and the EfficientNet-B4 scores and limits page follows one row across all ten sets.

06How realistic did people find the fakes?

The paper ran a user study with 100 participants, most of them computer vision researchers, who each rated 30 clips from each of seven datasets on a five-point scale, with 4 or 5 counted as "looks real" (section 3.4, Table 3). The share rated real was 64.1 percent for DeeperForensics-1.0, 61.0 percent for Celeb-DF, 23.0 percent for DFDC, 21.9 percent for the Google and Jigsaw DFD set, 14.1 percent for UADFV, 12.3 percent for DeepFake-TIMIT and 8.4 percent for FaceForensics++. The authors note that Celeb-DF reaches a similar realism score at a much smaller scale.

Those are ratings by trained viewers of clips chosen at random, not a detection task, and the participants were rating realism rather than guessing real or fake, so the figures are not comparable with the accuracy studies on the human studies page. The number the authors wanted was a quality ranking of datasets, and on that measure their set and Celeb-DF stood apart from the rest in 2020.

07Why numbers differ between sources

The paper describes "over 60,000" videos and "over 50,000" source videos; the DeepfakeBench table rounds to exactly 50,000 real and 10,000 fake, and this site uses the DeepfakeBench counts because they are the ones its results tables are built on. The DeepfakeBench README spells the set "DeepForensics-1.0"; the paper and this site use DeeperForensics-1.0. The 17.6 million frame total belongs to the whole set and the 12.6 million figure to the source footage alone. The paper's baselines are video-level accuracy on its hidden test set; DeepfakeBench's are frame-level AUC on the standard test split, so the two sets of scores measure different things and should not be placed in one column. Biometric evaluation convention, reporting one error rate at a fixed level of the other, is a third way of scoring; the biometric face recognition testing page sets it out.