What is a deepfake video, and how is one detected?

This page is the video entry in the deepfake detection hub. The broader what counts as a deepfake page covers stills and voices as well; this one stays with moving faces, because video is where the first datasets were built and where most published detector numbers come from.

01What is a deepfake video?

The term covers four kinds of alteration, in the taxonomy DF40 uses: face swapping, where one person's face replaces another's; face reenactment, where the target's own face is driven to new expressions or speech; entire face synthesis, where the face is generated from nothing; and face editing, where attributes such as age or hair are changed (arXiv 2406.13495). What makes it a video rather than a stack of edited photographs is that the alteration has to hold across frames, with the face tracking the head, the lighting and the mouth movements of the original clip.

FaceForensics++ (ICCV 2019), the dataset most detectors are trained on, contains the two oldest kinds. Its 1,000 source videos, taken from 977 YouTube clips, were manipulated with four methods: DeepFakes and FaceSwap, which swap the face, and Face2Face and NeuralTextures, which reenact it (arXiv 1901.08971). A later addition, FaceShifter, swapped the same 1,000 originals with a newer method, which is why it appears in detector tables as a separate test set.

02How is a deepfake video made?

The FaceForensics++ paper divides its four methods into two families. Face2Face and FaceSwap are graphics-based: Face2Face reenacts the target's expressions from a source actor, and FaceSwap transfers the face region. DeepFakes and NeuralTextures are learning-based: DeepFakes trains two autoencoders with a shared encoder, one per identity, and swaps the decoders; NeuralTextures modifies the expressions of the mouth region. In each case the altered face is produced for every frame and then placed back into the original footage.

That last step is the one most detectors of the FaceForensics++ era rely on. DF40 describes the common swap pipeline as creating a mask around the facial region and "blending the generated deepfake face back into the background image using that mask". The blending boundary is a trace the generator did not intend to leave, and the Face X-ray family of detectors is built to find it.

03What makes a video different from a still?

Three things, all of which a detector has to live with. First, a video is many frames, so a 1,000-video set becomes a very large image set: FaceForensics++ yields more than 1.8 million manipulated images. Second, every frame is compressed by a video codec, and FaceForensics++ is released at three H.264 settings, raw, constant rate factor 23 and constant rate factor 40, because compression removes the very artefacts a detector looks for; the paper's XceptionNet scores 99.26 percent accuracy on raw video, 95.73 percent at 23 and 81.00 percent at 40. Third, the frames are related in time, so an alteration that is invisible in one frame may flicker between frames, and a detector can be built to look at the sequence rather than at each picture.

Most benchmark detectors do not. DeepfakeBench states that it "currently adopts the frame level evaluation to build a fair basis for comparison among detectors", sampling 32 frames per video for training and testing (arXiv 2307.01426). The temporal signal is left to the smaller family of video detectors.

04How do detectors score a video?

A frame-level detector runs on each sampled face crop and returns a score per frame; the published AUC is computed over all frames in the test set, so a video contributes as many points to the curve as it has sampled frames. A video-level result aggregates those scores into one per clip, usually by averaging, before the curve is drawn. The two readings of the same model differ, and papers do not always say which they report, which is why the frame-level against video-level scoring page exists. The curve itself, and the AUC and equal error rate read from it, are defined on the detector scoring metrics page.

The pipeline before the score is the same as in face recognition: a face detector finds the face in the frame, landmarks align and crop it to a fixed size, and the crop goes to the network. DeepfakeBench applies one such pipeline to all nine of its datasets so that differences in cropping cannot pass for differences between detectors.

05How well do detectors score deepfake videos?

On the set they were trained on, very well. In DeepfakeBench's Table 3 every detector is trained on FaceForensics++ c23 and tested on the same split and on eight others; Xception scores 0.9637 frame-level AUC within the set and UCF, the best within-dataset detector, scores 0.9705. On sets the detector has not seen, the scores fall: Xception to 0.8341 on DeeperForensics-1.0, 0.8163 on DeepFakeDetection, 0.7365 on Celeb-DF v2, 0.7077 on DFDC and 0.6249 on FaceShifter, for a cross-dataset average of 0.7718. The best cross-dataset average among the 15 detectors is 0.7875, by SPSL. The in-dataset and cross-dataset AUC rows carry every detector and every set.

Fakes collected from the internet lower the numbers again. Deepfake-Eval-2024 gathered 45 hours of video from 88 websites during 2024, and the best off-the-shelf open-source video model, GenConViT, scored 0.63 AUC on it against 0.96 on the benchmark it was built for; across models, video AUC fell by about 50 percent (arXiv 2503.02857).

06Can people tell a deepfake video from a real one?

Less well than the detectors, on the material the detectors were trained on. The FaceForensics++ authors ran a user study with 204 participants: accuracy was 68.69 percent on raw video, 66.57 percent at constant rate factor 23 and 58.73 percent at 40, and the paper concludes that trained detectors "clearly outperform human observers". The comparison is on the same four manipulations the detectors had seen, so it says people are poor at this, not that detectors are reliable elsewhere. On still faces the human result is worse, and the synthetic faces and trust page shows raters trusting AI faces more than real ones.

07What a detection result means for one video

A score on a single clip is a weak piece of evidence, and the tables above say how weak: for a clip from a platform, several open-source detectors in 2024 sat near chance. The checking a video in order of evidence page puts a detector score after provenance and context for that reason, and the detector calibration page explains why a score of 0.9 is not a 90 percent probability unless it was calibrated on material like the clip in hand. No single benchmark score predicts how a detector performs on unseen manipulations.