Lip-sync deepfakes: how they are made and how detectors catch them

This page sits in the kinds-of-deepfake strand of the deepfake detection hub. The four kinds of face deepfake page places lip-sync inside face reenactment; this page follows one kind from generator to detector.

01What is a lip-sync deepfake?

In DF40's taxonomy a face reenactment "preserves its identity but has its intrinsic attributes, e.g., pose, mouth and expression manipulated" (arXiv 2406.13495). Lip-sync is the narrowest case: the driving signal is audio rather than a second face, and the only attribute changed is the mouth. The person in the output is the person in the input, saying something they did not say.

The Wav2Lip paper (ACM Multimedia 2020) states the task as "lip-syncing a talking face video of an arbitrary identity to match a target speech segment" (arXiv 2008.10010). Earlier methods, it notes, produced accurate lips on a static image or on people seen in training but drifted out of sync on unconstrained video. Its fix was to train the generator against a pre-trained lip-sync discriminator, and its abstract reports that the lip-sync accuracy of its output on the paper's own benchmarks "is almost as good as real synced videos".

02How is a lip-sync deepfake made?

The pipeline in the figure has three stages. A speech segment and the original video go in. A network renders the lower face frame by frame so that the mouth shape matches the audio at that instant, with a discriminator judging whether sound and lips agree. The rendered region is composited back into the original frame, so every pixel outside the mouth is the source video's own. That locality is what separates lip-sync from a face swap, which replaces the whole facial region, and from a full reenactment, which drives pose and expression as well.

The same locality appears inside FaceForensics++. Its NeuralTextures method learns a neural texture and a rendering network for each target and, in the authors' words, "We only modify the facial expressions corresponding to the mouth region" (arXiv 1901.08971). NeuralTextures is not audio-driven, but it is the mouth-only manipulation that frame-level detectors have found hardest: in DeepfakeBench Table 3 it is the lowest of the four FaceForensics++ manipulation columns for all 15 detectors.

03Which datasets contain lip-sync deepfakes?

Lip-sync fakes entered the public datasets later than swaps, and mostly through Wav2Lip. The table lists the sets on this site that name a lip-sync or audio-driven method, with the counts their papers give; the which deepfake dataset should you test on? table holds the rest of each set's description.

LAV-DF's abstract makes the point that lip-sync raises for detection: "a sophisticated deepfake may contain only a small segment of video/audio manipulation, through which the meaning of the content can be, for example, completely inverted" (arXiv 2204.06228). A whole-video label misses it; the task becomes finding where in time the mouth was changed.

04How do detectors catch lip-sync deepfakes?

LipForensics (CVPR 2021) is the detector built for this class. Its authors observe that "most face video forgeries alter the mouth in some way", that fake mouths "often do not adequately close when they pronounce certain phonemes", and that generated mouths show unnatural speed changes and frame-to-frame shape shifts even in silence (arXiv 2012.07657). The method pretrains a spatio-temporal network, a ResNet-18 with an initial 3D convolution followed by a temporal convolutional network, to lipread on the Lipreading in the Wild set, then fine-tunes the temporal part on real and fake mouth crops. The input is 25-frame clips of 96 by 96 greyscale mouth crops; predictions are averaged over the video.

The reasoning is that lipreading forces the network to learn what natural mouth motion looks like at a semantic level, so that a lip-sync fake fails not on texture but on movement. That cue survives compression and blur, which erase the pixel-level traces frame-based detectors depend on. The detection techniques compared page sets this temporal family beside the spatial and frequency ones; the MesoNet page shows how a 2018 frame-level network scores on the same mouth-only manipulation.

05What does the LipForensics paper report?

The paper's Table 2 trains each model on FaceForensics++ and scores video-level AUC, in percent, on four unseen sets. The figures below are the paper's own, on its own protocol: 110 test frames per video, predictions averaged over the whole clip, and baselines re-run or taken from their papers as the authors state. They are not comparable to frame-level AUC in the DeepfakeBench table, and the which detector scores highest on unseen fakes? page keeps the two protocols in separate rows for that reason.

Within FaceForensics++, trained on three manipulations and tested on the fourth, LipForensics scores 99.1 on NeuralTextures where Xception scores 79.7 (Table 1). FTCN, a later temporal detector that adopted the LipForensics protocol and copied its baseline rows, edges it on every set (arXiv 2108.06693, Table 4). Both papers report on sets built from swaps and reenactments; neither table contains a Wav2Lip set, so the published figures measure mouth-motion detection on fakes whose mouths changed for other reasons.

06What the audio adds

A lip-sync deepfake has two forged channels when the speech is synthetic too, and FakeAVCeleb labels them apart. Audio detection has its own benchmarks and its own generalisation gap, covered on the voice deepfake benchmarks page, where methods trained on ASVspoof 2019 lose up to one thousand percent in error rate on found recordings. A detector that reads both channels and checks that they agree is the natural next step, and LAV-DF's localisation task is the published form of that question. No single benchmark score predicts how a detector performs on unseen manipulations, and for lip-sync the public tables have not yet been built on the generator that makes most of them.