The Google and Jigsaw Deep Fake Detection dataset: contents, access and scores

This page is part of the deepfake detection hub. It sits beside the FaceForensics++ page, because the set is downloaded with the same script, under the same terms and in the same compression levels, and it is counted as a separate column in every results table.

01What is in the Deep Fake Detection dataset?

The FaceForensics dataset README describes it in one sentence: "The DeepFakeDetection dataset contains over 363 original sequences from 28 paid actors in 16 different scenes as well as over 3000 manipulated videos using DeepFakes" (FaceForensics repository). The download page itemises 363 original source actor videos and 3,068 manipulated videos, and the Celeb-DF paper's comparison table gives the same 363 and 3,068 with 28 subjects (arXiv 1909.12962).

The fakes are face swaps, manipulated real footage in the sense the deepfake versus AI-generated page sets out. The announcement states that "pairs of actors were selected randomly and deep neural networks swapped the face of one actor onto the head of another", using "publicly available deepfake generation methods". The repository names the manipulation as DeepFakes, the autoencoder swap also used for one of the four FaceForensics++ methods. DeepfakeBench's dataset table lists the set with 5 synthesis methods, and the Celeb-DF authors write that "the details of the synthesis algorithm are not disclosed", so the exact generator mix is not public.

02How was it recorded and released?

The set was announced on the Google Research blog on 24 September 2019, in collaboration with Jigsaw, as a contribution "incorporated into the Technical University of Munich and the University Federico II of Naples' new FaceForensics benchmark". The post says the team "worked with paid and consenting actors to record hundreds of videos" over the preceding year and then "created thousands of deepfakes from these videos". The FaceForensics README's changelog dates the addition a day earlier: "23.09.2019: Added sample videos as well as the Deep Fake Detection Dataset".

Consent is what sets the set apart from its neighbours. FaceForensics++ took its 1,000 sequences from 977 YouTube videos and Celeb-DF took 590 interviews of public figures from the same site; neither asked the people shown. The DeepfakeBench dataset table marks DFD "rights cleared: YES" and FaceForensics++ and Celeb-DF "NO". The consent in deepfake datasets page compares the consent basis and takedown route of every public set.

03How does it compare with the other 2019 sets?

The Celeb-DF paper places DFD in the second generation of deepfake datasets, with the DFDC preview and Celeb-DF, against the first generation of UADFV, DF-TIMIT and the FaceForensics++ DeepFakes split. Its Table 1 dates DFD to September 2019, the DFDC preview to October and Celeb-DF to November. The figure above gives the real and fake counts of the four 2019 face-swap sets from that table; DFD has the fewest originals and, at 3,068 fakes from 363 sources, about 8.5 fakes per original.

The DeeperForensics-1.0 paper counts the set as 3,431 videos from 28 actors, which is 363 plus 3,068, and its user study rated 21.9 percent of DFD clips as real, against 8.4 percent for FaceForensics++ and 61.0 percent for Celeb-DF (arXiv 2001.03024, Table 3). The full set of public counts, with source of faces and licence for each, is in the table of public deepfake sets.

04Files, naming and compression

The set lives inside the FaceForensics++ directory tree. Originals are stored in an actors folder with filenames of the form actor number, double underscore, scene name; manipulated videos sit under DeepFakeDetection and are named target actor, source actor, sequence name and an eight-character experiment id, because "some actor pairings have been recorded multiple times". Masks are provided directly from the DeepFake output and so are not rectangular like the FaceForensics++ DeepFakes masks.

Every video comes at the three FaceForensics++ compression levels. The README gives the sizes: the 363 originals are about 200 GB raw, 3 GB at c23 and 400 MB at c40; the 3,068 fakes are about 1.6 TB raw, 22 GB at c23 and 3 GB at c40. DeepfakeBench uses only the c23 release of FaceForensics++, DFD and FaceShifter, so published DFD scores from that benchmark are c23 scores.

05What do detectors score on it?

In DeepfakeBench's Table 3, with every detector trained on FaceForensics++ c23, the DFD column runs from 0.5481 frame-level AUC for Meso4 to 0.8163 for Xception, the best in that column (arXiv 2307.01426). Xception's 0.8163 on DFD is higher than its 0.7365 on Celeb-DF v2 and its 0.7077 on DFDC, which fits the user-study ordering: DFD fakes passed as real less often than Celeb-DF fakes. The detector scores on each test set carry the whole column.

The Celeb-DF paper's earlier evaluation, run with released weights rather than retrained models, gives the same shape; the reproducing a detection result page sets out why its numbers and DeepfakeBench's differ. Xception-c23 scored 85.9 percent frame-level AUC on DFD against 99.7 on the FaceForensics++ DeepFakes split and 65.3 on Celeb-DF (Table 4), and FWA scored 74.3 on DFD. The paper also used DFD for its visual quality comparison, computing Mask-SSIM on the 311 DFD videos for which it had exact frame correspondences.

06Access and terms

There is no separate application. The FaceForensics README states that the set "has a similar file structure and is downloaded by default together with the regular dataset", through the same Google form and download script as FaceForensics++, with the data under the FaceForensics Terms of Use. The download script takes DeepFakeDetection or DeepFakeDetection_original as its dataset argument and a compression flag of raw, c23 or c40.

The announcement promised that "since the field is moving quickly, we'll add to this dataset as deepfake technology evolves over time". The FaceForensics README's changelog records one later addition to the repository, FaceShifter on 15 July 2020, and no further DFD release. The Celeb-DF page describes the set that was built two months after DFD to be harder than it, and what makes a deepfake realistic lists the four changes to the swap that made it so.