What makes a realistic deepfake: resolution, blending and motion

This page is part of the deepfake detection hub. It explains the visual side of the face swap versus reenactment distinction: what a swap has to get right to pass, and how the benchmarks measured the cost to detectors each time it did.

01Why did the first deepfake datasets look fake?

The Celeb-DF paper lists the artefacts of the first generation of face swaps: "low-quality synthesized faces, visible splicing boundaries, color mismatch, visible parts of the original face" and temporal flicker. The basic method behind them, which the paper calls the DeepFake maker, trains one encoder shared across subjects with a dedicated decoder per identity, then warps the decoded face back onto the target and trims it with a mask built from facial landmarks. Each step can leave a trace, and in the early sets most of them did.

The paper groups the public sets into two generations. UADFV, DF-TIMIT and the DeepFakes split of FaceForensics++ are the first generation, released between November 2018 and January 2019; the Google Deep Fake Detection set, the DFDC preview and Celeb-DF itself are the second, released between September and November 2019 (Table 1). DF-TIMIT's fakes were synthesised at 64 by 64 and 128 by 128 pixels, which is the resolution problem in its plainest form.

02What did Celeb-DF change in the swap?

Four refinements, each aimed at one named artefact. The synthesised face is rendered at 256 by 256 pixels instead of the usual 64 or 128, using "encoder and decoder models with more layers and increased dimensions", so that resizing and rotating it onto the target damages it less. Colour mismatch between the swapped face and the surrounding skin is reduced by perturbing the colours of training faces at random and applying a colour transfer between the output and the target.

The mask is rebuilt. Earlier sets used rectangular masks or masks drawn directly from landmarks, which "leaves the boundaries of the mask visible"; Celeb-DF synthesises a face slightly larger than the original, then builds a smoothness mask from the eyebrow and lower-lip landmarks with interpolated points on the cheeks and chin. Flicker is reduced by filtering the landmark sequence over time with a Kalman smoother, so the swapped region does not jump from frame to frame. The resulting set holds 590 real YouTube interviews of 59 public figures and 5,639 swaps, about 13 seconds each at 30 frames per second; the Celeb-DF page covers its splits and access terms.

03How is visual quality measured?

The paper scores it with Mask-SSIM, the structural similarity between the head region of a fake frame and the same region of the original frame, on a 0 to 1 scale where higher is closer. The authors note the measure is only comparative, because an identity change lowers it for every set in the same way, and on that comparison Celeb-DF has the highest average of the seven sets in Table 2.

The other measure is people. A later user study in the DeeperForensics-1.0 paper showed 100 participants, most of them computer vision researchers, 30 clips from each set and counted a clip as passing when it was rated 4 or 5 on a five-point realism scale. FaceForensics++ clips passed 8.4 percent of the time, the Google Deep Fake Detection set 21.9 percent, DFDC 23.0 percent, Celeb-DF 61.0 percent and DeeperForensics-1.0 itself 64.1 percent (arXiv 2001.03024, Table 3). The FaceForensics++ authors' own 204-person study on their uncompressed clips had found 68.69 percent accuracy at telling real from fake, which fell to 58.73 percent at the c40 compression level.

04How much realism costs a detector

The Celeb-DF paper ran nine published detectors with their released weights over seven sets and reported frame-level AUC. Xception-c23, the FaceForensics++ authors' model trained on the c23 release, scored 99.7 percent on the FaceForensics++ DeepFakes split, 85.9 on the Google set, 72.2 on the DFDC preview and 65.3 on Celeb-DF (Table 4); the figure above gives all seven. Averaged across detectors, the second-generation sets came in below 70 percent AUC while the first generation sat around 80 percent, with DSP-FWA the best overall method at 87.4 percent.

DeepfakeBench retrained the same detector under one protocol in 2023 and found the same ordering. Xception trained on FaceForensics++ c23 reached 0.9637 frame-level AUC on the FaceForensics++ test split and 0.7365 on Celeb-DF v2; the best detector on that column, SPSL, reached 0.7650 (arXiv 2307.01426, Table 3). The two Xception numbers differ, 65.3 against 73.65, because the two groups trained their own copies of the network with different frames, splits and code; the DeepfakeBench cross-dataset rows record each with its conditions, and reproducing a detection result traces the gap item by item.

05Does compression undo the realism gain?

Partly, and unevenly. Table 5 of the Celeb-DF paper re-encodes the set with H.264 at constant rate factors 23 and 40 and reruns four detectors. FWA falls from 56.9 to 54.6 and 52.2 percent AUC, DSP-FWA from 64.6 to 57.7 and 47.2, while Xception-c23 holds at 65.3, 65.5 and 52.5 and Xception-c40 at 65.5, 65.4 and 59.4. Detectors trained on compressed video kept their score at c23 and lost it at c40; detectors trained on clean frames lost it at both.

Compression therefore acts on a realistic fake much as it acts on a crude one: it removes the fine traces a detector relies on, which helps the fake. The effect on the same detector across FaceForensics++'s raw, c23 and c40 releases is set out on the FaceForensics++ compression levels page.

06Where realism went after the swap

The refinements above improve a face swap, a manipulation of real footage in the sense the deepfake versus AI-generated page sets out. Fully generated faces take a different route to realism: a generator trained on 70,000 photographs produces a whole face with no blending boundary at all, and the question becomes whether the generator leaves its own regularities in the pixels. The GAN and diffusion face synthesis page covers that family and the detectors built for it.

The two routes meet in the recent datasets. DF40 (2024) generated its fakes on the real footage of FaceForensics++ and Celeb-DF with 40 methods, 12 of them full synthesis, and the counts for every public set, with the generation and method mix of each, sit in the counts for every public set. No single visual quality score predicts how a detector performs on a set it has not seen; the realism of the swap is one cause of the drop, and the generator and the compression are the others.