FaceForensics++: the four manipulation methods, compression levels and how detectors are scored on it

This page is the FaceForensics++ entry in this site's dataset comparison and part of the deepfake detection hub. It describes the set as its paper and repository describe it, and it explains how to read the scores that reference it.

01What FaceForensics++ contains

The repository README states the composition plainly: "a forensics dataset consisting of 1000 original video sequences that have been manipulated with four automated face manipulation methods: Deepfakes, Face2Face, FaceSwap and NeuralTextures. The data has been sourced from 977 youtube videos and all videos contain a trackable mostly frontal face without occlusions which enables automated tampering methods to generate realistic forgeries" (FaceForensics repository). Binary masks mark the manipulated region in each frame, so the data serves segmentation as well as classification, and the authors released 1,000 trained Deepfakes models so that others can generate more.

The paper (arXiv 1901.08971, ICCV 2019) frames the contribution as a benchmark rather than a pile of videos: "an automated benchmark for facial manipulation detection ... based on DeepFakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size. The benchmark is publicly available and contains a hidden test set as well as a database of over 1.8 million manipulated images." At release it was, by the authors' account, more than an order of magnitude larger than comparable public forgery datasets.

Two additions came later. In September 2019 the repository began hosting the Deep Fake Detection dataset contributed by Google and Jigsaw, over 3,000 manipulated videos of 28 actors filmed for the purpose. In July 2020 the authors added FaceShifter, a two-stage face swapping method published at CVPR 2020, applied to all 1,000 original videos.

02The four manipulation methods

Deepfakes

The autoencoder face swap that gave the whole category its name: a shared encoder with two decoders, one per identity, so that a face from one person can be decoded as the other. The repository ships the 1,000 trained models used to make this subset.

Face2Face

A facial reenactment method: the expressions of a source actor are transferred onto the target's face in the video while the target's identity is kept. It alters what the face does, not whose face it is.

FaceSwap

A graphics-based face swap that fits a 3D face model and transfers the face region with it, rather than learning the swap from data.

NeuralTextures

A learned rendering method that modifies the mouth region through neural textures; it was added as the fourth method before the ICCV version of the paper and the benchmark was updated to include it.

The split matters for detection: two of the four are face swaps (identity changes) and two are reenactments or regional edits (expression changes), so a detector that generalises across FaceForensics++'s own subsets is already crossing two kinds of forgery. The kinds of face deepfake page sets out that taxonomy.

03Compression levels and why c40 matters

Every video in FaceForensics++ is provided raw and at two H.264 compression settings, c23 and c40, the number being the constant rate factor, so c40 is the most compressed and the lowest quality. Most published training uses c23; DeepfakeBench, for instance, uses c23 only for FaceForensics++ and its FaceShifter and Deep Fake Detection subsets (DeepfakeBench repository).

Compression removes the fine inconsistencies many detectors depend on. In DeepfakeBench's table, the Xception baseline trained on c23 scores a frame-level AUC of 0.9637 on the c23 test split and 0.8261 on c40, with nothing else changed. A detector that is only ever tested at c23 has not been tested against the quality of video people actually share, which has usually been through several rounds of platform recompression.

04The benchmark protocol and the official server

The paper distinguishes the dataset from the benchmark. The benchmark is a fixed set of 1,000 images drawn across the methods at random compression and size, with hidden labels, evaluated through an automated server hosted by the authors. Submitting to it is the one way to get a FaceForensics++ score that no one could have tuned to, and the repository links the current server and image set. Scores quoted from the benchmark server and scores quoted from a paper's own test split of the public data are different measurements, and this site's results table labels which is which.

05Published results on FaceForensics++ and their conditions

The published results page carries detector scores with their train set, test set, compression and metric. For FaceForensics++ the pattern is consistent across the DeepfakeBench table: in-dataset frame-level AUC at c23 sits between about 0.94 and 0.97 for every serious method, with UCF highest at 0.9705 and Xception at 0.9637, and per-method subsets vary (Face X-ray scores 0.9872 on the Face2Face subset, for example). The paper itself found that detectors "clearly outperform human observers" on this data. How to read AUC, and why it is preferred to accuracy here, is on the metrics page.

06From FaceForensics++ to unseen sets

The scores above describe FaceForensics++ fakes. On Celeb-DF v2 the same Xception detector falls to 0.7365 and on DFDC to 0.7077; even the best cross-dataset method in the 2023 table averages 0.7875 on unseen sets. The cross-dataset generalisation page explains the drop and the Celeb-DF page describes the set that was built to cause it. The DF40 authors' critique, that FaceForensics++ "contains out-of-date forgery techniques from the past four years", is the reason newer sets exist.

07Access

The dataset is released under the FaceForensics Terms of Use; access is by a request form on the repository, after which the authors send a download script. The repository notes that unanswered requests usually mean a bouncing email address. Nothing from the set is hosted on this site.