Face swap deepfakes: how a face swap is made and how detectors catch it

This page belongs to the deepfake detection hub and is the swap half of the kinds of face deepfake page, which separates swaps from reenactment and from fully synthetic faces.

01What is a face swap deepfake?

A face swap transfers identity. The source provides who the face is; the target provides everything else: head pose, expression, gaze, skin tone under the scene's lighting, and the frame the face sits in. The SimSwap paper states the task as "transferring the identity of an arbitrary source face into an arbitrary target face while preserving the attributes of the target face" (arXiv 2106.06340). A reenactment deepfake does the opposite: it keeps the target's identity and changes the expression, which is why FaceForensics++ files Face2Face and NeuralTextures separately from its two swaps.

Three things make a swap a swap, and each one leaves a trace a detector can use. The identity comes from a face recognition embedding or an autoencoder trained on the source. The synthesized face is pasted into the target frame, so there is a seam. And the frame is then re-encoded as video, so the seam is softened by compression.

02How were the FaceForensics++ swaps made?

FaceForensics++ chose "two computer graphics-based approaches (Face2Face and FaceSwap) and two learning-based approaches (DeepFakes and NeuralTextures)" and applied each to 1,000 YouTube videos (arXiv 1901.08971). The two swaps work in different ways.

FaceSwap is the graphics method. In the paper's words it "is a graphics-based approach to transfer the face region from a source video to a target video. Based on sparse detected facial landmarks the face region is extracted". The method fits "a 3D template model using blendshapes", projects it back onto the target frame to match the detected landmarks, textures it from the source, and then "the rendered model is blended with the image and color correction is applied". No network is trained; the result is a textured mesh pasted over the target face.

DeepFakes is the learning method, and the one that gave the whole category its name. The paper describes it as "based on two autoencoders with a shared encoder that are trained to reconstruct training images of the source and the target face". A face detector crops and aligns both sets of faces. At synthesis time the shared encoder reads the target face and the source person's decoder reconstructs it, so the output has the target's pose and expression in the source's identity. That output is then "blended with the rest of the image using Poisson image editing". FaceForensics++ used the public faceswap repository with default parameters and published the 1,000 trained video-pair models alongside the data; the FaceForensics++ page has the counts and compression levels.

03What changed with FaceShifter and SimSwap?

The DeepFakes method needs one trained model per pair of people, FaceSwap needs a landmark fit for every frame, and both paste a masked region over the target. FaceShifter (CVPR 2020) and SimSwap (ACM Multimedia 2020) generate the swapped face inside a network that has already seen many identities, so one model swaps any source into any target.

FaceShifter is a two-stage framework. Its first stage uses an attributes encoder to read the target face at several levels of detail and a generator with Adaptive Attentional Denormalization layers to combine those attributes with the source identity. Its second stage, HEAR-Net, is "trained to recover anomaly regions in a self-supervised way without any manual annotations", which is how it handles hair, glasses and hands crossing the face (arXiv 1912.13457). FaceForensics++ added a FaceShifter manipulation of all 1,000 originals on 15 July 2020, so the method now sits in the same benchmark as the swaps it replaced.

SimSwap works at the feature level too. Its ID Injection Module "transfers the identity information of the source face into the target face at feature level", and its Weak Feature Matching Loss keeps the target's expression and gaze without an explicit attribute encoder (arXiv 2106.06340). The identity itself is supplied by a pretrained ArcFace recognition model, which is the clearest link between swapping and the face recognition pipeline: the same kind of embedding that verifies a passport photo tells the generator which identity to inject. The paper also scores swaps with recognition: on FaceForensics++ pairs, a separate recognition network retrieves the correct source identity for 77.65% of DeepFakes frames, 97.38% of FaceShifter frames and 92.83% of SimSwap frames (Table 1).

04What do all face swaps have in common?

The Face X-ray authors made the observation that organises detection: "most existing face manipulation methods share a common step: blending the altered face into an existing background image" (arXiv 1912.13458). Whether the inner face came from a textured mesh, an autoencoder or a feature-level generator, it was composited onto a frame from a different source, and the two regions carry different noise, colour and compression statistics. The boundary between them is the one artefact a swap cannot avoid.

A detector that learns the artefacts of one generator does not learn this. The Face X-ray paper's Table 1 trains both Xception and Face X-ray on the DeepFakes subset of FaceForensics++ alone and tests them on all four manipulations. Xception scores 99.38 AUC on DeepFakes but 75.05 on Face2Face, 49.13 on FaceSwap and 80.39 on NeuralTextures. Face X-ray scores 99.17, 94.14, 75.34 and 93.85 on the same four. On FaceSwap, the other swap method, Xception is at chance and Face X-ray is 26 points above it. The Xception as a deepfake detector page lists that model's scores under every published condition.

05How does Face X-ray detect a swap?

Face X-ray is "a greyscale image that reveals whether the input image can be decomposed into the blending of two images from different sources". For a real image it is blank; for a swapped one it draws the blending boundary. The network is trained on blended images the authors build themselves from pairs of real faces, "without fake images generated by any of the state-of-the-art face manipulation methods", so it never sees a generator's artefacts and cannot over-fit to them.

Trained on those blended images only, Face X-ray reaches 93.47 AUC on the Google and Jigsaw DFD set, 71.15 on the DFDC release of the time and 74.76 on Celeb-DF, against 87.86, 48.98 and 36.19 for Xception trained on FaceForensics++ (Table 2 of the paper). Adding FaceForensics++ fakes to the blended images lifts Face X-ray to 95.40, 80.92 and 80.58. These are the paper's own measurements in 2020, on the datasets as they stood then.

Values in percent from Face X-ray, Table 2 (arXiv 1912.13458), read 9 November 2026.

06Does the blending boundary still work on newer swaps?

Only partly, and the figures say by how much. In DeepfakeBench (NeurIPS 2023), where every detector is trained on FaceForensics++ c23 under one protocol, Face X-ray scores 0.9872 frame-level AUC on the Face2Face subset and 0.9871 on FaceSwap, the best of the 15 detectors on Face2Face, but 0.6553 on the FaceShifter set and 0.5531 on DeeperForensics-1.0, with a cross-dataset average of 0.6985 against Xception's 0.7718 (arXiv 2307.01426, Table 3). FaceShifter's second stage repairs the region around the pasted face, which is exactly the boundary the method looks for, and DeeperForensics applies heavy perturbations that disturb boundary statistics. The published detector results carry those rows with their conditions.

No single artefact survives every generator. The blending boundary is the most general one found so far for swaps, and the AI-generated faces page explains why it does not apply at all to a face that was synthesized whole, with no target frame to blend into.