Video-level deepfake detection: temporal cues and clip scoring

This page belongs to the deepfake detection hub and follows on from the frame-level versus video-level AUC page, which explains why one detector can carry two numbers. Here the subject is the detectors built to use time, what they look at, and what they score.

01What does video-level detection mean?

A deepfake video is a sequence of manipulated frames, and the earliest detectors treated it as a bag of images: classify each frame, then combine. The LipForensics paper fixed the combination rule that later papers adopted: "we compute video-level measures for fair comparison: we average the model predictions (each prediction corresponding to a frame or a video clip) across the entire video" (arXiv 2012.07657). Under that rule a frame classifier and a clip classifier are compared on the same footing, one score per video.

A temporal detector goes one step further and lets the network see several frames at once. LipForensics feeds 25-frame clips of 96 by 96 greyscale mouth crops; FTCN takes an "input video clip" of "32 frames each with 224×224 pixels" (arXiv 2108.06693). The network can then respond to how the face changes between frames, which a single frame cannot show. The metric definitions on the deepfake detection metrics page apply unchanged; only the unit being scored moves from frame to clip.

02Which temporal cues do the detectors use?

Three published detectors illustrate three different cues. LipForensics, "Lips Don't Lie" (CVPR 2021), targets "high-level semantic irregularities in mouth movements, which are common in many generated videos". Its feature extractor is a ResNet-18 with an initial 3D convolution, pretrained for lipreading on the Lipreading in the Wild set and then frozen, with a temporal network fine-tuned on the mouth embeddings. The authors' observation is that "fake mouths often do not adequately close when they pronounce certain phonemes", a cue the lip-sync page under related reading examines from the generation side.

FTCN, "Exploring Temporal Coherence for More General Video Face Forgery Detection" (ICCV 2021), removes most of the spatial signal on purpose: the authors "reduce the spatial convolution kernel size to 1, while maintaining the temporal convolution kernel size unchanged", so the 3D ResNet-50 backbone is forced to learn from how pixels change over time, and a temporal transformer sits on top. TALL, "Thumbnail Layout for Deepfake Video Detection" (ICCV 2023), takes the opposite route. Its abstract describes the method: "consecutive frames are masked in a fixed position in each frame to improve generalization, then resized to sub-images and rearranged into a pre-defined layout as the thumbnail" (arXiv 2307.07494). Each clip of 4 frames becomes one 224 by 224 thumbnail, and an ordinary image backbone, a Swin-B transformer in the reported TALL-Swin model, reads the sequence as spatial layout.

03How do video detectors score across datasets?

Each of the three papers trains on FaceForensics++ HQ and tests on Celeb-DF v2 (CDF), the DeepFake Detection Challenge set (DFDC), FaceShifter (FSh) and DeeperForensics (DFo), reporting video-level AUC in percent. LipForensics prints the baseline rows, FTCN copies them and adds its own, and TALL repeats both with its new rows, so the table below comes from a single chain of sources with identical conditions (TALL paper, Table 3, arXiv 2307.07494).

The Face X-ray row is printed in the LipForensics and FTCN tables at one decimal place; the others come from the TALL table at two. Two patterns hold across every row. DFDC is the hardest column for all six detectors, with the best score at 76.78, and the two datasets whose fakes keep the source footage's timing, FaceShifter and DeeperForensics, are where temporal detectors gain most, from 72.00 and 84.50 for Xception to above 97 for the sequence models. The published results by dataset page keeps the frame-level DeepfakeBench rows separate from these, because the two levels are not comparable.

In-dataset the gap narrows. TALL's Table 2 reports video-level AUC on FaceForensics++ HQ of 96.30 for Xception, 99.70 for LipForensics and 99.87 for TALL-Swin, and on the heavily compressed LQ release 89.30, 98.10 and 94.57, with the authors noting that TALL-Swin "gets unsatisfactory results with the LQ setting". A detector that reads motion can lose that motion to compression.

04What does DeepfakeBench v2 list?

The DeepfakeBench README states that the framework "supports a total of 36 detection methods (28 image detectors + 8 video detectors)" and names the video group as TALL, I3D, STIL, FTCN, X-CLIP, TimeTransformer and VideoMAE, with AltFreezing (CVPR 2023) added in the same update note (README). Four of those, I3D, X-CLIP, TimeTransformer and VideoMAE, are general video recognition models rather than purpose-built forgery detectors, which is itself a finding: a strong video model fine-tuned on fakes is now a baseline.

The README also lists "frame-level AUC, video-level AUC, ACC (fake and real), EER, PR, and AP" as supported metrics, while its printed results table still covers the original fifteen image detectors at frame level and says the authors "will update the below table soon". The DeepfakeBench page tracks that table; as of this page's read date no video-level rows for the eight video detectors have been published in it, so the cross-dataset figures above remain the papers' own.

05How do video detectors fare on fakes from the wild?

Deepfake-Eval-2024 tested FTCN off the shelf on fakes collected from the internet in 2024. Its Table 3 gives FTCN an AUC of 0.50 on the new set against 0.87 on its original test data, and GenConViT, a video model with a hybrid design, 0.63 against 0.96 (arXiv 2503.02857). Fine-tuning on 60 percent of the set and testing on the other 40 percent raised FTCN to 0.71 and GenConViT to 0.82 (Table 4). The temporal cue that generalises across four curated sets did not survive the move to unknown generators and platform re-encoding; the detection in the wild page sets those drops beside the image and audio models.

06Why is a clip score not a guarantee?

A sequence model can be attacked as a sequence. Hussain et al. (WACV 2021) report a 3D CNN detector scoring 91.74 percent accuracy on benign DFDC validation videos, and a "robust white-box" perturbation that drove the attack success rate on compressed adversarial videos to 100 percent, with a black-box version reaching 24.43 percent and its robust variant 51.02 percent (arXiv 2002.12749). The adversarial attacks on deepfake detectors page gives the full tables. No single video-level AUC predicts how a detector performs on unseen manipulations, and the same number says nothing about an adaptive attacker.