Face reenactment: Face2Face, NeuralTextures and talking heads explained

This page is part of the deepfake detection hub and expands one of the four kinds of face deepfake. Face swap, the other common kind, replaces identity; reenactment leaves it alone, which is why the two leave different traces and why a detector trained on one often misses the other.

01What is face reenactment?

Reenactment transfers a performance onto a face that is not performing it. The DF40 paper defines it by what changes: face reenactment on a target "preserves its identity i_t but has its intrinsic attributes a_t, e.g., pose, mouth and expression manipulated" (arXiv 2406.13495). FaceForensics++ calls the same thing facial expression manipulation, one of the two families into which it sorts all face forgery; the other is identity manipulation, "known as face swapping" (arXiv 1901.08971).

The result looks like the real person because it is the real person. Only the motion is borrowed, so the face recognition pipeline that would flag a swapped identity has nothing to flag, and the evidence a detector can use is confined to the parts that were regenerated: the mouth, the expression wrinkles, and the join between the rendered region and the untouched frame.

02How does Face2Face transfer an expression?

Face2Face is the graphics based original. Its authors describe "a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video)" where "the source sequence is also a monocular video stream, captured live with a commodity webcam" (Face2Face project page). The method first recovers the target's identity "by non-rigid model-based bundling", then tracks the expressions of both actors with "a dense photometric consistency measure", and transfers them: "Reenactment is then achieved by fast and efficient deformation transfer between source and target".

The hardest part is the inside of the mouth, which a face model does not contain. Face2Face solves it by retrieval: "The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit", after which the synthesised face is re-rendered to match the real-world illumination of the target frame, in the paper's words. FaceForensics++ lists Face2Face with FaceSwap as one of its "two computer graphics-based approaches", against the "two learning-based approaches (DeepFakes and NeuralTextures)". No neural network generates the pixels; a 3D face model does, so the traces it leaves are the model's, at the mouth and at the blend.

03How does NeuralTextures differ?

NeuralTextures is the learned counterpart in the same benchmark. The FaceForensics++ paper states that it modifies mouth-region expressions, rendering the result from a learned neural texture of the target rather than from a fitted 3D model, and classes it with DeepFakes as learning based (arXiv 1901.08971). The FaceForensics++ dataset holds 1,000 videos of each, so the two reenactment methods can be compared on the same source footage.

They are not equally easy to catch. In DeepfakeBench Table 3, Xception trained on all of FaceForensics++ c23 reaches a frame-level AUC of 0.9785 on the Face2Face split and 0.9385 on the NeuralTextures split, the lowest of the four manipulations (arXiv 2307.01426). The same paper notes that in cross-manipulation tests "the 'FT-NT' test data poses challenges for almost all detectors". Face X-ray, which looks only for a blending boundary, scores 0.9872 on Face2Face, the best figure in that column, because a graphics method blends a rendered face into the frame. The DeepfakeBench results and their conditions page gives the full table.

AUC values are frame-level, from DeepfakeBench Table 3, detectors trained on FaceForensics++ c23 and tested on each manipulation's own split.

04What are talking heads, and how many does DF40 count?

A talking head is reenactment driven by something other than a second face. DF40's Table 2 lists 13 face reenactment methods, numbered 11 to 23, and labels how each is driven: FOMM, the First Order Motion Model, is image driven; FS_vid2vid is landmark driven; Wav2Lip and SadTalker are audio driven; HeyGen is text driven (arXiv 2406.13495). Most are generated on FaceForensics++ and Celeb-DF source footage at 1,500 or more videos each; HeyGen appears as 50 test-only clips because a one-to-one commercial pipeline takes "significantly more time and resources".

The two driving papers describe the mechanics. The First Order Motion Model animates "an object in a source image ... according to the motion of a driving video" using "a set of learned keypoints along with their local affine transformations", with no annotation of the face (arXiv 2003.00196). Wav2Lip addresses "lip-syncing a talking face video of an arbitrary identity to match a target speech segment" by "learning from a powerful lip-sync discriminator", and reports that its lip-sync accuracy "is almost as good as real synced videos" (arXiv 2008.10010). The generation methods page on this site lists all 40 DF40 techniques by type.

05How well do detectors generalise across reenactment methods?

Better than across kinds, with one exception. DF40's Protocol 1 trains a detector on one kind and tests it on another within the FaceForensics++ domain: Xception trained on the 13 reenactment methods scores 0.996 on reenactment, 0.838 on face swap and 0.670 on entire face synthesis (arXiv 2406.13495). The paper's Finding 5 states that "FR forgeries may share transferable patterns for detecting other FR instances", and names the exception: "Wav2Lip, an audio-driven reenactment forgery that only modifies the mouth region". A detector that learned whole-face motion artefacts has less to work with when only the lips moved.

Finding 1 records the asymmetry in the other direction: "model training on FS yields higher results (around 0.8) on EFS, while training on EFS only achieves about 0.6 on FS". Reenactment sits between: trained on face swap, Xception scores 0.892 on reenactment, trained on synthesis, 0.807. The research definitions of deepfake page explains why a Celeb-DF score, which covers swaps only, says nothing about any of these numbers.

06Where does reenactment meet detection in practice?

Reenactment is the kind that identity verification has to worry about most, because it is the kind a live attacker can drive. The ENISA remote identity proofing report records that "Deepfake videos and interactive deepfake puppets can now be created in real-time", and a puppet is a reenactment: an attacker's expressions on a victim's face, fed to the camera or injected behind it. The passive and active liveness checks page sets out what each kind of check can and cannot prove against it.

The law does not make the distinction. None of the eight statutes in the state deepfake statutes table separates a moved face from a swapped one; Texas defines deep fake media as any depiction of a real person "performing an action that did not occur in reality", which covers both. For a detector the distinction is everything: one benchmark finding, repeated in FaceForensics++, DeepfakeBench and DF40, is that the mouth region of a reenacted face is the hardest part of the hardest kind.