Deepfake videos: how they are made, scored and benchmarked

This page is part of the deepfake detection hub and takes the video case on its own. The kinds of face deepfake page separates swap, reenactment, editing and full synthesis; here the question is what a video adds to each, and how the shared benchmarks score it.

01What is a deepfake video?

The FaceForensics++ paper (ICCV 2019) sets the vocabulary. It divides facial manipulation into two categories, "facial expression manipulation and facial identity manipulation", and notes that "the term Deepfakes has widely become a synonym for face replacement based on deep learning" while also being the name of one specific method spread through online forums (arXiv 1901.08971).

DF40 (2024) widens the list to four types, face swapping, face reenactment, entire face synthesis and face editing, and builds forty techniques across them. Of its forty methods, the swap and reenactment families are the ones that produce video from video; the synthesis and editing methods in its table work on still images, and a text-driven video generator, HeyGen, is included under reenactment as a test-only set. The videos the benchmarks below score are swaps and reenactments.

02How are deepfake videos made?

The original DeepFakes method, as FaceForensics++ describes it, "is based on two autoencoders with a shared encoder that are trained to reconstruct training images of the source and the target face". At render time the encoder reads the target's face and the source's decoder redraws it, frame after frame, and each result is "blended with the rest of the image using Poisson image editing". The paper notes that a pair of models has to be trained for each pair of people, "which is a time-consuming task". FaceSwap, the second swap method in the set, is graphics-based: it fits a 3D template to sparse landmarks and blends the rendered face with colour correction.

Reenactment works on movement rather than identity. Face2Face transfers "76 Blendshape coefficients" of expression from the source to the target in each frame; NeuralTextures learns a neural texture and rendering network and "only modif[ies] the facial expressions corresponding to the mouth region". All four methods "require source and target actor video pairs as input", and "the final output of each method is a video composed of generated images". That last sentence is the one that matters for detection: a deepfake video is a sequence of edited frames that has then been encoded as video, so compression touches every frame of the evidence. Later generators changed the model inside the loop, with identity encoders borrowed from face recognition and, in DF40's list, diffusion models, but not the frame-by-frame structure; deepfake technology covers the model families.

03How are deepfake videos detected?

The standard pipeline, in FaceForensics++ and in DeepfakeBench after it, is per frame. A face tracker finds the face, a conservative crop around it is cut out, and a classifier returns a real or fake score for that crop. DeepfakeBench "currently adopts the frame level evaluation to build a fair basis for comparison among detectors", samples 32 frames per video for training and testing, and reports the area under the ROC curve over those frames (arXiv 2307.01426). The families of cue a frame detector can use, and what each scored under equal conditions, are set out in detection techniques compared.

Video-level detection scores the clip rather than the frame. It can average frame scores, or it can run a model over a stack of frames and look for inconsistency across time. DeepfakeBench lists video-level detectors and metrics as future work; DF40 reports one, I3D, by video-level AUC next to its frame-level detectors; Deepfake-Eval-2024 reports video models such as GenConViT on its video set.

04How are deepfake videos benchmarked?

Every benchmark is a set of real clips, a set of fakes made or found, and a protocol for which clips train and which test. The main video sets differ most in where the fakes came from.

Counts are as the papers state them; the datasets table in this section carries the rest of each row. The split between made and found fakes is the one that explains the scores below: a made set shares its generators and its compression between training and test, a found set shares nothing.

05What do the published scores say?

Within a benchmark, video detection looks solved. In DeepfakeBench Table 3, UCF scores 0.9705 frame-level AUC on the FaceForensics++ c23 test split it was trained on, and ten of fifteen detectors average above 0.93 across the FaceForensics++ splits. On eight datasets none of them saw, the best average is SPSL's 0.7875, and the naive Xception falls from 0.9637 at home to 0.7365 on Celeb-DF v2 and 0.7077 on DFDC. The published results table lists every detector by test set.

On fakes found rather than made, the numbers fall again. The DFDC paper reported that its winning model, trained only on the challenge set, kept a ROC-AUC of 0.734 and an average precision of 0.753 on a small set of real in-the-wild deepfakes the organisers collected (arXiv 2006.07397). Five years later, Deepfake-Eval-2024 found open-source detectors' AUC down by 50 percent for video against their original benchmarks; the best off-the-shelf open-source video model, GenConViT, scored 0.63 after scoring 0.96 on its own data, and the best of the anonymous commercial video models scored 0.79 (arXiv 2503.02857). No single benchmark score predicts how a detector performs on unseen manipulations; the ladder in the figure above is the evidence.

06Why does compression matter so much for video?

Because the video container re-encodes every frame, and the traces a detector learned may not survive. FaceForensics++ released the same fakes at three H.264 settings: raw, a constant rate quantisation parameter of 23 (its c23 or high quality release) and 40 (c40, low quality). XceptionNet trained on all four methods scored an accuracy of 99.26 percent on raw frames, 95.73 at c23 and 81.00 at c40, on identical content (Table 1). DeepfakeBench's Xception, trained at c23, scores 0.9637 AUC at c23 and 0.8261 at c40.

Circulating videos are re-encoded at least once by the platform that hosts them, often after a screen recording or a crop, so a detector's published c23 number is an upper bound for what it sees online.

07Can people tell a deepfake video from a real one?

Not reliably, and less so as quality drops. The FaceForensics++ user study asked 204 participants to judge single frames with a time limit of 2, 4 or 6 seconds: accuracy was 68.69 percent on raw frames, 66.57 at c23 and 58.73 at c40, against 50 percent chance. In Groh et al. (PNAS 2022), recruited participants shown whole DFDC clips were right in 66 percent of attempts, and the leading DFDC model scored 65 percent on the full 4,000-video holdout set. Which cues people can and cannot see, with the per-method numbers, are on the signs of a deepfake page.