How do deepfakes work? The pipeline from source clip to finished output

This page sits in the deepfake detection hub as the mechanism behind the definition of a deepfake. It follows one pipeline through the published method sections; the kinds of face deepfake page sorts the outputs into categories, and this one explains how each category is produced.

01What goes into a deepfake?

Every method in the FaceForensics++ dataset starts from a pair of videos: "All four methods require source and target actor video pairs as input", and the output "is a video composed of generated images" (Rössler et al., 2019). The dataset's 1,000 source videos came from 977 YouTube clips. The paper separates manipulation methods into two families, "facial expression manipulation and facial identity manipulation": an identity swap needs footage of the person whose face will be inserted, while a reenactment needs a source performance to copy.

The amount of footage the model needs is the first thing that changed. The DeepFakes method must be trained on both faces: "DeepFakes need to be trained for each pair of videos, which is a timeconsuming task". FaceShifter (CVPR 2020) and SimSwap (ACM Multimedia 2020) replaced the per-pair training with an identity encoder trained once, so a single source image can be swapped into any target clip. SimSwap's authors describe the older design directly: "Once trained, it was able to swap faces between two specific identities but lacked the ability for generalization" (Chen et al., 2020).

02How is the face found and aligned?

No generator works on a whole frame. A face detector locates the face, landmarks fix its geometry, and the crop is rotated and scaled so that the eyes and mouth land in standard positions. The FaceForensics++ description of DeepFakes states that "A face detector is used to crop and to align the images" before training; SimSwap's implementation details say the authors "align and crop the images to a standard position with size 224 × 224". The same step opens every detector pipeline, and the crop defines the region that will later be pasted back, whose edge is where blending artefacts live.

03How does a face swap autoencoder work?

The DeepFakes method, in the FaceForensics++ paper's words, is "based on two autoencoders with a shared encoder that are trained to reconstruct training images of the source and the target face". One encoder compresses any face into a code; two decoders, one per person, learn to rebuild their own person's face from that code. At render time "the trained encoder and decoder of the source face are applied to the target face": the target's frame is encoded, and the source person's decoder turns the code into the source person's face wearing the target's expression and pose.

The WildDeepfake paper describes the same architecture as it appears in open-source tools, "one encoder and two decoders", and notes that the swapped face is usually fused into the frame with an attention mask (Zi et al., 2020). The face swap deepfakes page covers the swap family in depth; the point here is that the swap is a decoder substitution, not an edit of pixels.

04What changed with identity encoders?

FaceShifter and SimSwap keep the encoder and decoder idea but split what the network is told about each face. FaceShifter uses "a pretrained state-of-the-art face recognition model" as its identity encoder, a separate multi-level attributes encoder for everything about the target that is not identity, and a generator whose Adaptive Attentional Denormalization layers decide, region by region, whether to follow the identity or the attributes; a second stage, HEAR-Net, repairs occlusions such as glasses and reflections "in a self-supervised way without any manual annotations" (Li et al., 2020). SimSwap injects an ArcFace identity vector into the target's features through its ID Injection Module so that "the relevance between identity information and the weights of Decoder can be removed and our architecture can be applied to arbitrary identities", and adds a Weak Feature Matching Loss to keep the target's expression and gaze.

The DF40 benchmark (2024) formalises the split into a DF-family, which "involves creating a mask around the facial region ... and blending the generated deepfake face back into the background image using that mask", and an FS-family that uses an identity-background encoder and generates the whole image, background included (Yan et al., 2024). FaceForensics++, Celeb-DF and DFDC belong to the first family.

05How is reenactment different?

Reenactment keeps the target's identity and changes what the face does. Face2Face, the first of the two reenactment methods in FaceForensics++, fits a 3D face model to the target, tracks the source actor's expression as 76 blendshape coefficients per frame, and re-renders the target's face with those coefficients. NeuralTextures learns a neural texture of the target plus a rendering network, trained with a photometric reconstruction loss and an adversarial loss, and in the dataset "we only modify the facial expressions corresponding to the mouth region". FaceSwap, the graphics-based swap in the same dataset, fits a 3D template with blendshapes in the same way, then renders and blends the source face.

06How is the face put back and encoded?

The generated face is a tight crop. To make a frame, it is pasted into the original image and the seam is hidden: DeepFakes output is "blended with the rest of the image using Poisson image editing", FaceSwap applies colour correction, and the DF40 DF-family definition is this blend step. The frames are then re-encoded as video. FaceForensics++ releases three versions of every clip: raw, and two H.264 encodings at constant rate factor 23 and 40, which the release calls c23 and c40.

Compression is the step that most changes what a detector or a viewer can see. In the FaceForensics++ user study, 204 participants identified manipulated clips at 68.69 percent on raw video, 66.57 percent at c23 and 58.73 percent at c40; the XceptionNet classifier trained on all four methods scored 99.26, 95.73 and 81.00 percent on the same three versions.

07Where do fully generated faces fit?

A generated face has no source clip at all. StyleGAN2 (Karras et al., 2020) is the generator behind the synthetic faces used in the human studies this site covers: trained on the 70,000 FFHQ images at 1024 by 1024, it reaches a Fréchet inception distance of 2.84 against 4.40 for the original StyleGAN in the paper's Table 1 (arXiv 1912.04958). The same paper shows how pipelines leave traces: its authors found that instance normalisation "causes water droplet -like artifacts in StyleGAN images" and redesigned the generator to remove them, and the new generator "becomes significantly easier to invert", which "makes it possible to reliably attribute a generated image to a particular network". DF40 lists 12 entire face synthesis methods, GAN and diffusion based, as the third of its four families.

08What does the pipeline leave behind?

Each step is a place where a detector can look: a boundary around the pasted region, frequency patterns from the generator's upsampling layers, and the fine detail that compression removes from real and fake alike. A detector trained on one dataset learns the traces of that dataset's pipeline. In the DeepfakeBench table, Xception trained on FaceForensics++ c23 reaches a frame-level AUC of 0.9637 on that dataset and 0.7365 on Celeb-DF v2, whose swaps were made with a different pipeline (arXiv 2307.01426); the AUC score explained page says what those two numbers mean. On fakes collected from the internet, where the pipeline is unknown, image detectors that score above 95 percent on FaceForensics++ fall below 70 percent, as the WildDeepfake dataset page records.