FaceShifter: the face swap method added to FaceForensics++ in 2020, and how detectors score on it

This page sits in the deepfake detection hub beside the FaceForensics++ page, which covers the four original manipulations. FaceShifter is the fifth, added eighteen months after the dataset's release, and the only one of the five that DeepfakeBench scores as a cross-domain set rather than a within-domain split.

01What is FaceShifter?

FaceShifter swaps the identity of a source face onto a target face in two stages. The first stage, which the paper calls AEI-Net, has three parts: an identity encoder, a multi-level attributes encoder, and a generator built from Adaptive Attentional Denormalization (AAD) layers that decide, at each feature level, where to use identity information and where to keep the target's attributes (arXiv 1912.13457). The identity encoder is not trained from scratch: the authors "use a pretrained state-of-the-art face recognition model as identity encoder", so the swap is steered by the same kind of embedding a recognition system compares. The face recognition terms that describe such embeddings, template and similarity score, apply to the generator's own loss.

The second stage, HEAR-Net, handles occlusions such as hair or an object in front of the face. It is "trained to recover anomaly regions in a self-supervised way without any manual annotations", using the difference between a face and its own reconstruction as a signal for where the first stage went wrong. Faces are aligned and cropped to 256 by 256 pixels, and AEI-Net is trained on CelebA-HQ, FFHQ and VGGFace. The method is a swap, not a reenactment; the face swap versus reenactment page explains why that distinction matters for detection.

02How did FaceShifter get into FaceForensics++?

The FaceForensics repository's changelog records "15.07.2020: Added FaceShifter", after the Deep Fake Detection Dataset entry of 23 September 2019 and the original "25.01.2019: Release of FaceForensics++" (GitHub README). The README states that "All 1000 original videos of the original youtube based dataset have been manipulated", and the dataset's own page says the videos were manipulated by the FaceShifter authors and shared with the FaceForensics team. The repository also notes that, compared with the four earlier methods, FaceShifter can "deal with facial occlusions using a second synthesis stage".

DeepfakeBench lists FaceShifter as a dataset of its own: 1,000 real and 1,000 fake videos, 2,000 in all, one synthesis method, rights not cleared, and, like FaceForensics++ and DFD, used at the c23 compression level only (DeepfakeBench README). The public dataset rows with counts carry the same numbers next to the other sets.

03How well does FaceShifter preserve identity?

The paper measures identity preservation with an ID retrieval test on FaceForensics++: for each swapped face, a separate face recognition model searches all original frames for the nearest face and checks whether it belongs to the correct source video. FaceShifter scores 97.38 percent on this test, against 81.96 for DeepFakes, 82.41 for IPGAN, 76.57 for the method of Nirkin et al. and 54.19 for FaceSwap (Table 1). Pose error, the L2 distance between head pose vectors of the swapped and target faces, is 2.96, and expression error 2.06, the lowest of the five methods for expression. The test set is 10,000 face images per method, ten frames sampled evenly from each clip.

A user study with 100 evaluators asked which of four outputs best kept the source identity, best kept the target's attributes, and looked most real. FaceShifter was chosen 52.9, 62.1 and 82.9 percent of the time respectively (Table 2). High identity fidelity is the point of the method, and it is also what makes the set hard for detectors that learned the blending traces of weaker swaps.

04How do detectors score on FaceShifter?

DeepfakeBench trains every detector on FaceForensics++ c23 and tests it on eight unseen sets, FaceShifter among them, reporting frame-level AUC (arXiv 2307.01426, Table 3). The column is low for every method. Face X-ray leads at 0.6553, followed by Capsule 0.6465, UCF 0.6462, MesoInception 0.6438 and SPSL 0.6437; the Xception baseline scores 0.6249, EfficientNet-B4 0.6162, and DSP-FWA 0.5551. For comparison, Xception scores 0.9379 on UADFV and 0.7365 on Celeb-DF v2 in the same table, so FaceShifter is its weakest cross-domain set by a clear margin.

That Face X-ray tops the column fits its design: it looks for the blending boundary that any swap leaves, rather than for the artefacts of a particular generator, and the Face X-ray detector page shows the same pattern on other swaps. The UCF detector, which has the best in-dataset score in the table, is third here at 0.6462, a reminder that in-dataset and cross-dataset rankings differ. The cross-dataset rows for the FaceShifter column sit with their conditions next to every other test set.

05Why numbers differ between sources

Two kinds of difference appear in the literature. The first is protocol. LipForensics (CVPR 2021) reports video-level AUC on the FaceShifter HQ set after training on FaceForensics++: Xception 72.0 percent, Face X-ray 92.8 and LipForensics 97.1 (arXiv 2012.07657, Table 2). DeepfakeBench's 0.6249 for Xception is frame-level, uses its own retrained model and its own preprocessing, so the two Xception values are not the same measurement. Video-level scores aggregate many frames and are usually higher.

The second is re-implementation. The SimSwap paper repeats FaceShifter's ID retrieval test with its own runs and reports DeepFakes at 77.65 percent where FaceShifter's paper gives 81.96; FaceShifter itself is listed at 97.38 in both (arXiv 2106.06340). A score for a method depends on who ran it and on which recognition model scored the retrieval.

06Why FaceShifter matters for benchmarks

FaceShifter shows what happens when the generator improves after the detector was trained. The four 2019 manipulations in FaceForensics++ are detected at 0.9385 or better frame-level AUC by Xception when the test split comes from the same set; the 2020 swap, built from the same 1,000 source videos, drops the same model to 0.6249. The source footage is identical, the compression is identical, and only the manipulation changed. That isolates the generator as the cause in a way that a test on a different dataset cannot.

It also shows the direction later generators took. FaceShifter's identity signal comes from a pretrained recognition network, and SimSwap follows the same recipe with an ArcFace model. A detector that learned the blending seams of a graphics-based swap has little to go on when the identity is injected inside the network, which is one reason a swap of this kind is the usual payload in the deepfake attack routes that reach a camera or a video call.